{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Continuous Control\n",
    "\n",
    "---\n",
    "\n",
    "Congratulations for completing the second project of the [Deep Reinforcement Learning Nanodegree](https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893) program!  In this notebook, you will learn how to control an agent in a more challenging environment, where the goal is to train a creature with four arms to walk forward.  **Note that this exercise is optional!**\n",
    "\n",
    "### 1. Start the Environment\n",
    "\n",
    "We begin by importing the necessary packages.  If the code cell below returns an error, please revisit the project instructions to double-check that you have installed [Unity ML-Agents](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Installation.md) and [NumPy](http://www.numpy.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from unityagents import UnityEnvironment\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will start the environment!  **_Before running the code cell below_**, change the `file_name` parameter to match the location of the Unity environment that you downloaded.\n",
    "\n",
    "- **Mac**: `\"path/to/Crawler.app\"`\n",
    "- **Windows** (x86): `\"path/to/Crawler_Windows_x86/Crawler.exe\"`\n",
    "- **Windows** (x86_64): `\"path/to/Crawler_Windows_x86_64/Crawler.exe\"`\n",
    "- **Linux** (x86): `\"path/to/Crawler_Linux/Crawler.x86\"`\n",
    "- **Linux** (x86_64): `\"path/to/Crawler_Linux/Crawler.x86_64\"`\n",
    "- **Linux** (x86, headless): `\"path/to/Crawler_Linux_NoVis/Crawler.x86\"`\n",
    "- **Linux** (x86_64, headless): `\"path/to/Crawler_Linux_NoVis/Crawler.x86_64\"`\n",
    "\n",
    "For instance, if you are using a Mac, then you downloaded `Crawler.app`.  If this file is in the same folder as the notebook, then the line below should appear as follows:\n",
    "```\n",
    "env = UnityEnvironment(file_name=\"Crawler.app\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\t\n",
      "Unity brain name: CrawlerBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 129\n",
      "        Number of stacked Vector Observation: 1\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): 20\n",
      "        Vector Action descriptions: , , , , , , , , , , , , , , , , , , , \n"
     ]
    }
   ],
   "source": [
    "env = UnityEnvironment(file_name='Crawler.exe')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the State and Action Spaces\n",
    "\n",
    "Run the code cell below to print some information about the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 12\n",
      "Size of each action: 20\n",
      "There are 12 agents. Each observes a state with length: 129\n",
      "The state for the first agent looks like: [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  2.25000000e+00\n",
      "  1.00000000e+00  0.00000000e+00  1.78813934e-07  0.00000000e+00\n",
      "  1.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  6.06093168e-01 -1.42857209e-01 -6.06078804e-01  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  1.33339906e+00 -1.42857209e-01\n",
      " -1.33341408e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      " -6.06093347e-01 -1.42857209e-01 -6.06078625e-01  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00 -1.33339953e+00 -1.42857209e-01\n",
      " -1.33341372e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      " -6.06093168e-01 -1.42857209e-01  6.06078804e-01  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00 -1.33339906e+00 -1.42857209e-01\n",
      "  1.33341408e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  6.06093347e-01 -1.42857209e-01  6.06078625e-01  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  1.33339953e+00 -1.42857209e-01\n",
      "  1.33341372e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00]\n"
     ]
    }
   ],
   "source": [
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "\n",
    "# number of agents\n",
    "num_agents = len(env_info.agents)\n",
    "print('Number of agents:', num_agents)\n",
    "\n",
    "# size of each action\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Size of each action:', action_size)\n",
    "\n",
    "# examine the state space \n",
    "states = env_info.vector_observations\n",
    "state_size = states.shape[1]\n",
    "print('There are {} agents. Each observes a state with length: {}'.format(states.shape[0], state_size))\n",
    "print('The state for the first agent looks like:', states[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Take Random Actions in the Environment\n",
    "\n",
    "In the next code cell, you will learn how to use the Python API to control the agent and receive feedback from the environment.\n",
    "\n",
    "Once this cell is executed, you will watch the agent's performance, if it selects an action at random with each time step.  A window should pop up that allows you to observe the agent, as it moves through the environment.  \n",
    "\n",
    "Of course, as part of the project, you'll have to change the code so that the agent is able to use its experience to gradually choose better actions when interacting with the environment!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#env_info = env.reset(train_mode=False)[brain_name]     # reset the environment    \n",
    "#states = env_info.vector_observations                  # get the current state (for each agent)\n",
    "#scores = np.zeros(num_agents)                          # initialize the score (for each agent)\n",
    "#while True:\n",
    "#    actions = np.random.randn(num_agents, action_size) # select an action (for each agent)\n",
    "#    actions = np.clip(actions, -1, 1)                  # all actions between -1 and 1\n",
    "#    env_info = env.step(actions)[brain_name]           # send all actions to tne environment\n",
    "#    next_states = env_info.vector_observations         # get next state (for each agent)\n",
    "#    rewards = env_info.rewards                         # get reward (for each agent)\n",
    "#    dones = env_info.local_done                        # see if episode finished\n",
    "#    scores += env_info.rewards                         # update the score (for each agent)\n",
    "#    states = next_states                               # roll over states to next time step\n",
    "#    if np.any(dones):                                  # exit loop if episode finished\n",
    "#        break\n",
    "#print('Total score (averaged over agents) this episode: {}'.format(np.mean(scores)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When finished, you can close the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. It's Your Turn!\n",
    "\n",
    "Now it's your turn to train your own agent to solve the environment!  When training the environment, set `train_mode=True`, so that the line for resetting the environment looks like the following:\n",
    "```python\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from  collections  import deque\n",
    "from itertools import count\n",
    "import torch\n",
    "import time\n",
    "#from ddpg_agent import Agent\n",
    "from ppo_agent import Agent\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "agent = Agent(state_size=state_size, action_size=action_size, random_seed=8,\\\n",
    "              n_agent=num_agents, fc1_units=1024, fc2_units=1024)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 1, Score: 49.02, Max: 49.02, Min: 49.02, T-max: 1000   \n",
      "Episode: 2, Score: 47.64, Max: 49.02, Min: 47.64, T-max: 1000   \n",
      "Episode: 3, Score: 48.36, Max: 49.02, Min: 47.64, T-max: 1000   \n",
      "Episode: 4, Score: 48.19, Max: 49.02, Min: 47.64, T-max: 1000   \n",
      "Episode: 5, Score: 48.85, Max: 49.02, Min: 47.64, T-max: 1000   \n",
      "Episode: 6, Score: 48.28, Max: 49.02, Min: 47.64, T-max: 1000   \n",
      "Episode: 7, Score: 51.84, Max: 51.84, Min: 47.64, T-max: 1000   \n",
      "Episode: 8, Score: 79.07, Max: 79.07, Min: 47.64, T-max: 1499   \n",
      "Episode: 9, Score: 77.57, Max: 79.07, Min: 47.64, T-max: 1499   \n",
      "Episode: 10, Score: 48.18, Max: 79.07, Min: 47.64, T-max: 1000   \n",
      "*** Episode 10 \t Average Score (over agents): 48.18 \t Average Score on 100 Episode: 54.70, Time: 00:02:21***\n",
      "Episode: 11, Score: 51.89, Max: 79.07, Min: 47.64, T-max: 1000   \n",
      "Episode: 12, Score: 54.78, Max: 79.07, Min: 47.64, T-max: 1000   \n",
      "Episode: 13, Score: 57.38, Max: 79.07, Min: 47.64, T-max: 1000   \n",
      "Episode: 14, Score: 58.13, Max: 79.07, Min: 47.64, T-max: 1000   \n",
      "Episode: 15, Score: 87.51, Max: 87.51, Min: 47.64, T-max: 1499   \n",
      "Episode: 16, Score: 88.17, Max: 88.17, Min: 47.64, T-max: 1499   \n",
      "Episode: 17, Score: 90.89, Max: 90.89, Min: 47.64, T-max: 1499   \n",
      "Episode: 18, Score: 64.11, Max: 90.89, Min: 47.64, T-max: 1000   \n",
      "Episode: 19, Score: 62.11, Max: 90.89, Min: 47.64, T-max: 1000   \n",
      "Episode: 20, Score: 65.40, Max: 90.89, Min: 47.64, T-max: 1000   \n",
      "*** Episode 20 \t Average Score (over agents): 65.40 \t Average Score on 100 Episode: 61.37, Time: 00:04:57***\n",
      "Episode: 21, Score: 67.59, Max: 90.89, Min: 47.64, T-max: 1000   \n",
      "Episode: 22, Score: 66.06, Max: 90.89, Min: 47.64, T-max: 1000   \n",
      "Episode: 23, Score: 64.93, Max: 90.89, Min: 47.64, T-max: 1000   \n",
      "Episode: 24, Score: 66.51, Max: 90.89, Min: 47.64, T-max: 1000   \n",
      "Episode: 25, Score: 67.80, Max: 90.89, Min: 47.64, T-max: 1000   \n",
      "Episode: 26, Score: 65.86, Max: 90.89, Min: 47.64, T-max: 1000   \n",
      "Episode: 27, Score: 107.94, Max: 107.94, Min: 47.64, T-max: 1499   \n",
      "Episode: 28, Score: 76.87, Max: 107.94, Min: 47.64, T-max: 1000   \n",
      "Episode: 29, Score: 69.61, Max: 107.94, Min: 47.64, T-max: 1000   \n",
      "Episode: 30, Score: 76.42, Max: 107.94, Min: 47.64, T-max: 1000   \n",
      "*** Episode 30 \t Average Score (over agents): 76.42 \t Average Score on 100 Episode: 65.23, Time: 00:07:17***\n",
      "Episode: 31, Score: 118.07, Max: 118.07, Min: 47.64, T-max: 1499   \n",
      "Episode: 32, Score: 78.57, Max: 118.07, Min: 47.64, T-max: 1000   \n",
      "Episode: 33, Score: 78.88, Max: 118.07, Min: 47.64, T-max: 1000   \n",
      "Episode: 34, Score: 81.46, Max: 118.07, Min: 47.64, T-max: 1000   \n",
      "Episode: 35, Score: 83.18, Max: 118.07, Min: 47.64, T-max: 1000   \n",
      "Episode: 36, Score: 85.08, Max: 118.07, Min: 47.64, T-max: 1000   \n",
      "Episode: 37, Score: 122.83, Max: 122.83, Min: 47.64, T-max: 1499   \n",
      "Episode: 38, Score: 126.88, Max: 126.88, Min: 47.64, T-max: 1499   \n",
      "Episode: 39, Score: 91.69, Max: 126.88, Min: 47.64, T-max: 1000   \n",
      "Episode: 40, Score: 135.29, Max: 135.29, Min: 47.64, T-max: 1499   \n",
      "*** Episode 40 \t Average Score (over agents): 135.29 \t Average Score on 100 Episode: 73.97, Time: 00:10:08***\n",
      "Episode: 41, Score: 134.05, Max: 135.29, Min: 47.64, T-max: 1499   \n",
      "Episode: 42, Score: 148.53, Max: 148.53, Min: 47.64, T-max: 1499   \n",
      "Episode: 43, Score: 151.20, Max: 151.20, Min: 47.64, T-max: 1499   \n",
      "Episode: 44, Score: 155.66, Max: 155.66, Min: 47.64, T-max: 1499   \n",
      "Episode: 45, Score: 152.76, Max: 155.66, Min: 47.64, T-max: 1499   \n",
      "Episode: 46, Score: 158.00, Max: 158.00, Min: 47.64, T-max: 1499   \n",
      "Episode: 47, Score: 160.57, Max: 160.57, Min: 47.64, T-max: 1499   \n",
      "Episode: 48, Score: 108.59, Max: 160.57, Min: 47.64, T-max: 1000   \n",
      "Episode: 49, Score: 110.12, Max: 160.57, Min: 47.64, T-max: 1000   \n",
      "Episode: 50, Score: 162.40, Max: 162.40, Min: 47.64, T-max: 1499   \n",
      "*** Episode 50 \t Average Score (over agents): 162.40 \t Average Score on 100 Episode: 88.02, Time: 00:13:48***\n",
      "Episode: 51, Score: 172.04, Max: 172.04, Min: 47.64, T-max: 1499   \n",
      "Episode: 52, Score: 173.98, Max: 173.98, Min: 47.64, T-max: 1499   \n",
      "Episode: 53, Score: 116.20, Max: 173.98, Min: 47.64, T-max: 1000   \n",
      "Episode: 54, Score: 119.78, Max: 173.98, Min: 47.64, T-max: 1000   \n",
      "Episode: 55, Score: 173.26, Max: 173.98, Min: 47.64, T-max: 1499   \n",
      "Episode: 56, Score: 123.28, Max: 173.98, Min: 47.64, T-max: 1000   \n",
      "Episode: 57, Score: 124.97, Max: 173.98, Min: 47.64, T-max: 1000   \n",
      "Episode: 58, Score: 121.48, Max: 173.98, Min: 47.64, T-max: 1000   \n",
      "Episode: 59, Score: 197.04, Max: 197.04, Min: 47.64, T-max: 1499   \n",
      "Episode: 60, Score: 123.75, Max: 197.04, Min: 47.64, T-max: 1000   \n",
      "*** Episode 60 \t Average Score (over agents): 123.75 \t Average Score on 100 Episode: 97.44, Time: 00:16:25***\n",
      "Episode: 61, Score: 190.13, Max: 197.04, Min: 47.64, T-max: 1499   \n",
      "Episode: 62, Score: 193.34, Max: 197.04, Min: 47.64, T-max: 1499   \n",
      "Episode: 63, Score: 198.82, Max: 198.82, Min: 47.64, T-max: 1499   \n",
      "Episode: 64, Score: 137.64, Max: 198.82, Min: 47.64, T-max: 1000   \n",
      "Episode: 65, Score: 136.28, Max: 198.82, Min: 47.64, T-max: 1000   \n",
      "Episode: 66, Score: 145.25, Max: 198.82, Min: 47.64, T-max: 1000   \n",
      "Episode: 67, Score: 223.75, Max: 223.75, Min: 47.64, T-max: 1499   \n",
      "Episode: 68, Score: 148.48, Max: 223.75, Min: 47.64, T-max: 1000   \n",
      "Episode: 69, Score: 227.38, Max: 227.38, Min: 47.64, T-max: 1499   \n",
      "Episode: 70, Score: 167.45, Max: 227.38, Min: 47.64, T-max: 1000   \n",
      "*** Episode 70 \t Average Score (over agents): 167.45 \t Average Score on 100 Episode: 108.79, Time: 00:19:15***\n",
      "Episode: 71, Score: 162.96, Max: 227.38, Min: 47.64, T-max: 1000   \n",
      "Episode: 72, Score: 249.47, Max: 249.47, Min: 47.64, T-max: 1499   \n",
      "Episode: 73, Score: 278.68, Max: 278.68, Min: 47.64, T-max: 1499   \n",
      "Episode: 74, Score: 274.87, Max: 278.68, Min: 47.64, T-max: 1499   \n",
      "Episode: 75, Score: 194.76, Max: 278.68, Min: 47.64, T-max: 1000   \n",
      "Episode: 76, Score: 197.00, Max: 278.68, Min: 47.64, T-max: 1000   \n",
      "Episode: 77, Score: 299.34, Max: 299.34, Min: 47.64, T-max: 1499   \n",
      "Episode: 78, Score: 203.69, Max: 299.34, Min: 47.64, T-max: 1000   \n",
      "Episode: 79, Score: 201.76, Max: 299.34, Min: 47.64, T-max: 1000   \n",
      "Episode: 80, Score: 205.07, Max: 299.34, Min: 47.64, T-max: 1000   \n",
      "*** Episode 80 \t Average Score (over agents): 205.07 \t Average Score on 100 Episode: 123.53, Time: 00:21:55***\n",
      "Episode: 81, Score: 209.57, Max: 299.34, Min: 47.64, T-max: 1000   \n",
      "Episode: 82, Score: 211.42, Max: 299.34, Min: 47.64, T-max: 1000   \n",
      "Episode: 83, Score: 216.64, Max: 299.34, Min: 47.64, T-max: 1000   \n",
      "Episode: 84, Score: 332.34, Max: 332.34, Min: 47.64, T-max: 1499   \n",
      "Episode: 85, Score: 218.38, Max: 332.34, Min: 47.64, T-max: 1000   \n",
      "Episode: 86, Score: 224.00, Max: 332.34, Min: 47.64, T-max: 1000   \n",
      "Episode: 87, Score: 239.95, Max: 332.34, Min: 47.64, T-max: 1000   \n",
      "Episode: 88, Score: 343.10, Max: 343.10, Min: 47.64, T-max: 1499   \n",
      "Episode: 89, Score: 344.36, Max: 344.36, Min: 47.64, T-max: 1499   \n",
      "Episode: 90, Score: 384.19, Max: 384.19, Min: 47.64, T-max: 1499   \n",
      "*** Episode 90 \t Average Score (over agents): 384.19 \t Average Score on 100 Episode: 140.07, Time: 00:24:33***\n",
      "Episode: 91, Score: 389.42, Max: 389.42, Min: 47.64, T-max: 1499   \n",
      "Episode: 92, Score: 261.46, Max: 389.42, Min: 47.64, T-max: 1000   \n",
      "Episode: 93, Score: 272.89, Max: 389.42, Min: 47.64, T-max: 1000   \n",
      "Episode: 94, Score: 249.03, Max: 389.42, Min: 47.64, T-max: 1000   \n",
      "Episode: 95, Score: 375.70, Max: 389.42, Min: 47.64, T-max: 1499   \n",
      "Episode: 96, Score: 249.76, Max: 389.42, Min: 47.64, T-max: 1000   \n",
      "Episode: 97, Score: 255.23, Max: 389.42, Min: 47.64, T-max: 1000   \n",
      "Episode: 98, Score: 232.07, Max: 389.42, Min: 47.64, T-max: 1000   \n",
      "Episode: 99, Score: 204.12, Max: 389.42, Min: 47.64, T-max: 1000   \n",
      "Episode: 100, Score: 234.52, Max: 389.42, Min: 47.64, T-max: 1000   \n",
      "*** Episode 100 \t Average Score (over agents): 234.52 \t Average Score on 100 Episode: 153.31, Time: 00:27:07***\n",
      "Episode: 101, Score: 254.82, Max: 389.42, Min: 47.64, T-max: 1000   \n",
      "Episode: 102, Score: 392.98, Max: 392.98, Min: 47.64, T-max: 1499   \n",
      "Episode: 103, Score: 267.88, Max: 392.98, Min: 47.64, T-max: 1000   \n",
      "Episode: 104, Score: 363.95, Max: 392.98, Min: 47.64, T-max: 1499   \n",
      "Episode: 105, Score: 372.40, Max: 392.98, Min: 47.64, T-max: 1499   \n",
      "Episode: 106, Score: 352.54, Max: 392.98, Min: 47.64, T-max: 1499   \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 107, Score: 247.34, Max: 392.98, Min: 47.64, T-max: 1000   \n",
      "Episode: 108, Score: 192.55, Max: 392.98, Min: 47.64, T-max: 1000   \n",
      "Episode: 109, Score: 411.90, Max: 411.90, Min: 47.64, T-max: 1499   \n",
      "Episode: 110, Score: 287.68, Max: 411.90, Min: 47.64, T-max: 1000   \n",
      "*** Episode 110 \t Average Score (over agents): 287.68 \t Average Score on 100 Episode: 179.28, Time: 00:29:55***\n",
      "Episode: 111, Score: 410.27, Max: 411.90, Min: 47.64, T-max: 1499   \n",
      "Episode: 112, Score: 258.51, Max: 411.90, Min: 47.64, T-max: 1000   \n",
      "Episode: 113, Score: 230.68, Max: 411.90, Min: 47.64, T-max: 1000   \n",
      "Episode: 114, Score: 311.35, Max: 411.90, Min: 47.64, T-max: 1499   \n",
      "Episode: 115, Score: 395.29, Max: 411.90, Min: 47.64, T-max: 1499   \n",
      "Episode: 116, Score: 250.51, Max: 411.90, Min: 47.64, T-max: 1000   \n",
      "Episode: 117, Score: 410.50, Max: 411.90, Min: 47.64, T-max: 1499   \n",
      "Episode: 118, Score: 244.56, Max: 411.90, Min: 47.64, T-max: 1000   \n",
      "Episode: 119, Score: 266.94, Max: 411.90, Min: 47.64, T-max: 1000   \n",
      "Episode: 120, Score: 257.52, Max: 411.90, Min: 47.64, T-max: 1000   \n",
      "*** Episode 120 \t Average Score (over agents): 257.52 \t Average Score on 100 Episode: 202.84, Time: 00:32:34***\n",
      "Episode: 121, Score: 235.39, Max: 411.90, Min: 47.64, T-max: 1000   \n",
      "Episode: 122, Score: 210.03, Max: 411.90, Min: 47.64, T-max: 1000   \n",
      "Episode: 123, Score: 262.14, Max: 411.90, Min: 47.64, T-max: 1000   \n",
      "Episode: 124, Score: 369.56, Max: 411.90, Min: 47.64, T-max: 1499   \n",
      "Episode: 125, Score: 326.66, Max: 411.90, Min: 47.64, T-max: 1499   \n",
      "Episode: 126, Score: 377.46, Max: 411.90, Min: 47.64, T-max: 1499   \n",
      "Episode: 127, Score: 335.64, Max: 411.90, Min: 47.64, T-max: 1499   \n",
      "Episode: 128, Score: 387.97, Max: 411.90, Min: 47.64, T-max: 1499   \n",
      "Episode: 129, Score: 398.52, Max: 411.90, Min: 47.64, T-max: 1499   \n",
      "Episode: 130, Score: 373.69, Max: 411.90, Min: 47.64, T-max: 1499   \n",
      "*** Episode 130 \t Average Score (over agents): 373.69 \t Average Score on 100 Episode: 228.31, Time: 00:35:35***\n",
      "Episode: 131, Score: 400.13, Max: 411.90, Min: 47.64, T-max: 1499   \n",
      "Episode: 132, Score: 387.52, Max: 411.90, Min: 47.64, T-max: 1499   \n",
      "Episode: 133, Score: 401.14, Max: 411.90, Min: 47.64, T-max: 1499   \n",
      "Episode: 134, Score: 354.87, Max: 411.90, Min: 47.64, T-max: 1499   \n",
      "Episode: 135, Score: 403.76, Max: 411.90, Min: 47.64, T-max: 1499   \n",
      "Episode: 136, Score: 367.48, Max: 411.90, Min: 47.64, T-max: 1499   \n",
      "Episode: 137, Score: 254.97, Max: 411.90, Min: 47.64, T-max: 1000   \n",
      "Episode: 138, Score: 384.14, Max: 411.90, Min: 47.64, T-max: 1499   \n",
      "Episode: 139, Score: 392.56, Max: 411.90, Min: 47.64, T-max: 1499   \n",
      "Episode: 140, Score: 412.63, Max: 412.63, Min: 47.64, T-max: 1499   \n",
      "*** Episode 140 \t Average Score (over agents): 412.63 \t Average Score on 100 Episode: 255.88, Time: 00:38:58***\n",
      "Episode: 141, Score: 419.27, Max: 419.27, Min: 47.64, T-max: 1499   \n",
      "Episode: 142, Score: 439.07, Max: 439.07, Min: 47.64, T-max: 1499   \n",
      "Episode: 143, Score: 180.64, Max: 439.07, Min: 47.64, T-max: 1499   \n",
      "Episode: 144, Score: 199.96, Max: 439.07, Min: 47.64, T-max: 1499   \n",
      "Episode: 145, Score: 454.93, Max: 454.93, Min: 47.64, T-max: 1499   \n",
      "Episode: 146, Score: 450.78, Max: 454.93, Min: 47.64, T-max: 1499   \n",
      "Episode: 147, Score: 403.27, Max: 454.93, Min: 47.64, T-max: 1499   \n",
      "Episode: 148, Score: 404.04, Max: 454.93, Min: 47.64, T-max: 1499   \n",
      "Episode: 149, Score: 436.66, Max: 454.93, Min: 47.64, T-max: 1499   \n",
      "Episode: 150, Score: 427.63, Max: 454.93, Min: 47.64, T-max: 1499   \n",
      "*** Episode 150 \t Average Score (over agents): 427.63 \t Average Score on 100 Episode: 279.63, Time: 00:42:21***\n",
      "Episode: 151, Score: 412.30, Max: 454.93, Min: 47.64, T-max: 1499   \n",
      "Episode: 152, Score: 397.05, Max: 454.93, Min: 47.64, T-max: 1499   \n",
      "Episode: 153, Score: 423.08, Max: 454.93, Min: 47.64, T-max: 1499   \n",
      "Episode: 154, Score: 422.12, Max: 454.93, Min: 47.64, T-max: 1499   \n",
      "Episode: 155, Score: 439.66, Max: 454.93, Min: 47.64, T-max: 1499   \n",
      "Episode: 156, Score: 452.11, Max: 454.93, Min: 47.64, T-max: 1499   \n",
      "Episode: 157, Score: 441.12, Max: 454.93, Min: 47.64, T-max: 1499   \n",
      "Episode: 158, Score: 451.08, Max: 454.93, Min: 47.64, T-max: 1499   \n",
      "Episode: 159, Score: 455.10, Max: 455.10, Min: 47.64, T-max: 1499   \n",
      "Episode: 160, Score: 415.96, Max: 455.10, Min: 47.64, T-max: 1499   \n",
      "*** Episode 160 \t Average Score (over agents): 415.96 \t Average Score on 100 Episode: 308.26, Time: 00:45:43***\n",
      "Episode: 161, Score: 460.54, Max: 460.54, Min: 47.64, T-max: 1499   \n",
      "Episode: 162, Score: 468.21, Max: 468.21, Min: 47.64, T-max: 1499   \n",
      "Episode: 163, Score: 500.15, Max: 500.15, Min: 47.64, T-max: 1499   \n",
      "Episode: 164, Score: 456.63, Max: 500.15, Min: 47.64, T-max: 1499   \n",
      "Episode: 165, Score: 516.33, Max: 516.33, Min: 47.64, T-max: 1499   \n",
      "Episode: 166, Score: 498.61, Max: 516.33, Min: 47.64, T-max: 1499   \n",
      "Episode: 167, Score: 502.33, Max: 516.33, Min: 47.64, T-max: 1499   \n",
      "Episode: 168, Score: 501.22, Max: 516.33, Min: 47.64, T-max: 1499   \n",
      "Episode: 169, Score: 509.42, Max: 516.33, Min: 47.64, T-max: 1499   \n",
      "Episode: 170, Score: 517.06, Max: 517.06, Min: 47.64, T-max: 1499   \n",
      "*** Episode 170 \t Average Score (over agents): 517.06 \t Average Score on 100 Episode: 339.88, Time: 00:49:01***\n",
      "Episode: 171, Score: 481.90, Max: 517.06, Min: 47.64, T-max: 1499   \n",
      "Episode: 172, Score: 501.21, Max: 517.06, Min: 47.64, T-max: 1499   \n",
      "Episode: 173, Score: 517.49, Max: 517.49, Min: 47.64, T-max: 1499   \n",
      "Episode: 174, Score: 516.91, Max: 517.49, Min: 47.64, T-max: 1499   \n",
      "Episode: 175, Score: 499.68, Max: 517.49, Min: 47.64, T-max: 1499   \n",
      "Episode: 176, Score: 494.22, Max: 517.49, Min: 47.64, T-max: 1499   \n",
      "Episode: 177, Score: 527.15, Max: 527.15, Min: 47.64, T-max: 1499   \n",
      "Episode: 178, Score: 514.31, Max: 527.15, Min: 47.64, T-max: 1499   \n",
      "Episode: 179, Score: 514.38, Max: 527.15, Min: 47.64, T-max: 1499   \n",
      "Episode: 180, Score: 500.72, Max: 527.15, Min: 47.64, T-max: 1499   \n",
      "*** Episode 180 \t Average Score (over agents): 500.72 \t Average Score on 100 Episode: 367.89, Time: 00:52:25***\n",
      "Episode: 181, Score: 520.18, Max: 527.15, Min: 47.64, T-max: 1499   \n",
      "Episode: 182, Score: 539.80, Max: 539.80, Min: 47.64, T-max: 1499   \n",
      "Episode: 183, Score: 535.90, Max: 539.80, Min: 47.64, T-max: 1499   \n",
      "Episode: 184, Score: 534.66, Max: 539.80, Min: 47.64, T-max: 1499   \n",
      "Episode: 185, Score: 519.81, Max: 539.80, Min: 47.64, T-max: 1499   \n",
      "Episode: 186, Score: 485.55, Max: 539.80, Min: 47.64, T-max: 1499   \n",
      "Episode: 187, Score: 535.56, Max: 539.80, Min: 47.64, T-max: 1499   \n",
      "Episode: 188, Score: 530.00, Max: 539.80, Min: 47.64, T-max: 1499   \n",
      "Episode: 189, Score: 536.44, Max: 539.80, Min: 47.64, T-max: 1499   \n",
      "Episode: 190, Score: 518.07, Max: 539.80, Min: 47.64, T-max: 1499   \n",
      "*** Episode 190 \t Average Score (over agents): 518.07 \t Average Score on 100 Episode: 393.21, Time: 00:55:42***\n",
      "Episode: 191, Score: 520.97, Max: 539.80, Min: 47.64, T-max: 1499   \n",
      "Episode: 192, Score: 537.15, Max: 539.80, Min: 47.64, T-max: 1499   \n",
      "Episode: 193, Score: 508.78, Max: 539.80, Min: 47.64, T-max: 1499   \n",
      "Episode: 194, Score: 520.61, Max: 539.80, Min: 47.64, T-max: 1499   \n",
      "Episode: 195, Score: 541.17, Max: 541.17, Min: 47.64, T-max: 1499   \n",
      "Episode: 196, Score: 524.68, Max: 541.17, Min: 47.64, T-max: 1499   \n",
      "Episode: 197, Score: 516.33, Max: 541.17, Min: 47.64, T-max: 1499   \n",
      "Episode: 198, Score: 539.20, Max: 541.17, Min: 47.64, T-max: 1499   \n",
      "Episode: 199, Score: 553.59, Max: 553.59, Min: 47.64, T-max: 1499   \n",
      "Episode: 200, Score: 552.23, Max: 553.59, Min: 47.64, T-max: 1499   \n",
      "*** Episode 200 \t Average Score (over agents): 552.23 \t Average Score on 100 Episode: 419.11, Time: 00:59:00***\n",
      "Episode: 201, Score: 574.55, Max: 574.55, Min: 47.64, T-max: 1499   \n",
      "Episode: 202, Score: 578.95, Max: 578.95, Min: 47.64, T-max: 1499   \n",
      "Episode: 203, Score: 565.38, Max: 578.95, Min: 47.64, T-max: 1499   \n",
      "Episode: 204, Score: 557.81, Max: 578.95, Min: 47.64, T-max: 1499   \n",
      "Episode: 205, Score: 569.28, Max: 578.95, Min: 47.64, T-max: 1499   \n",
      "Episode: 206, Score: 575.83, Max: 578.95, Min: 47.64, T-max: 1499   \n",
      "Episode: 207, Score: 581.68, Max: 581.68, Min: 47.64, T-max: 1499   \n",
      "Episode: 208, Score: 566.57, Max: 581.68, Min: 47.64, T-max: 1499   \n",
      "Episode: 209, Score: 568.19, Max: 581.68, Min: 47.64, T-max: 1499   \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 210, Score: 560.09, Max: 581.68, Min: 47.64, T-max: 1499   \n",
      "*** Episode 210 \t Average Score (over agents): 560.09 \t Average Score on 100 Episode: 444.66, Time: 01:02:33***\n",
      "Episode: 211, Score: 573.60, Max: 581.68, Min: 47.64, T-max: 1499   \n",
      "Episode: 212, Score: 597.32, Max: 597.32, Min: 47.64, T-max: 1499   \n",
      "Episode: 213, Score: 550.64, Max: 597.32, Min: 47.64, T-max: 1499   \n",
      "Episode: 214, Score: 588.04, Max: 597.32, Min: 47.64, T-max: 1499   \n",
      "Episode: 215, Score: 577.29, Max: 597.32, Min: 47.64, T-max: 1499   \n",
      "Episode: 216, Score: 590.64, Max: 597.32, Min: 47.64, T-max: 1499   \n",
      "Episode: 217, Score: 578.64, Max: 597.32, Min: 47.64, T-max: 1499   \n",
      "Episode: 218, Score: 570.09, Max: 597.32, Min: 47.64, T-max: 1499   \n",
      "Episode: 219, Score: 597.21, Max: 597.32, Min: 47.64, T-max: 1499   \n",
      "Episode: 220, Score: 597.89, Max: 597.89, Min: 47.64, T-max: 1499   \n",
      "*** Episode 220 \t Average Score (over agents): 597.89 \t Average Score on 100 Episode: 472.51, Time: 01:05:54***\n",
      "Episode: 221, Score: 579.83, Max: 597.89, Min: 47.64, T-max: 1499   \n",
      "Episode: 222, Score: 576.34, Max: 597.89, Min: 47.64, T-max: 1499   \n",
      "Episode: 223, Score: 614.83, Max: 614.83, Min: 47.64, T-max: 1499   \n",
      "Episode: 224, Score: 621.99, Max: 621.99, Min: 47.64, T-max: 1499   \n",
      "Episode: 225, Score: 596.29, Max: 621.99, Min: 47.64, T-max: 1499   \n",
      "Episode: 226, Score: 592.06, Max: 621.99, Min: 47.64, T-max: 1499   \n",
      "Episode: 227, Score: 607.17, Max: 621.99, Min: 47.64, T-max: 1499   \n",
      "Episode: 228, Score: 603.61, Max: 621.99, Min: 47.64, T-max: 1499   \n",
      "Episode: 229, Score: 597.14, Max: 621.99, Min: 47.64, T-max: 1499   \n",
      "Episode: 230, Score: 590.80, Max: 621.99, Min: 47.64, T-max: 1499   \n",
      "*** Episode 230 \t Average Score (over agents): 590.80 \t Average Score on 100 Episode: 499.54, Time: 01:09:13***\n",
      "Episode: 231, Score: 611.60, Max: 621.99, Min: 47.64, T-max: 1499   \n",
      "Episode: 232, Score: 598.81, Max: 621.99, Min: 47.64, T-max: 1499   \n",
      "Episode: 233, Score: 605.33, Max: 621.99, Min: 47.64, T-max: 1499   \n",
      "Episode: 234, Score: 608.57, Max: 621.99, Min: 47.64, T-max: 1499   \n",
      "Episode: 235, Score: 604.67, Max: 621.99, Min: 47.64, T-max: 1499   \n",
      "Episode: 236, Score: 609.10, Max: 621.99, Min: 47.64, T-max: 1499   \n",
      "Episode: 237, Score: 587.44, Max: 621.99, Min: 47.64, T-max: 1499   \n",
      "Episode: 238, Score: 592.30, Max: 621.99, Min: 47.64, T-max: 1499   \n",
      "Episode: 239, Score: 604.31, Max: 621.99, Min: 47.64, T-max: 1499   \n",
      "Episode: 240, Score: 595.03, Max: 621.99, Min: 47.64, T-max: 1499   \n",
      "*** Episode 240 \t Average Score (over agents): 595.03 \t Average Score on 100 Episode: 522.12, Time: 01:12:44***\n",
      "Episode: 241, Score: 621.28, Max: 621.99, Min: 47.64, T-max: 1499   \n",
      "Episode: 242, Score: 615.32, Max: 621.99, Min: 47.64, T-max: 1499   \n",
      "Episode: 243, Score: 574.26, Max: 621.99, Min: 47.64, T-max: 1499   \n",
      "Episode: 244, Score: 560.52, Max: 621.99, Min: 47.64, T-max: 1499   \n",
      "Episode: 245, Score: 616.04, Max: 621.99, Min: 47.64, T-max: 1499   \n",
      "Episode: 246, Score: 622.05, Max: 622.05, Min: 47.64, T-max: 1499   \n",
      "Episode: 247, Score: 255.88, Max: 622.05, Min: 47.64, T-max: 1499   \n",
      "Episode: 248, Score: 143.97, Max: 622.05, Min: 47.64, T-max: 1499   \n",
      "Episode: 249, Score: 557.49, Max: 622.05, Min: 47.64, T-max: 1499   \n",
      "Episode: 250, Score: 571.67, Max: 622.05, Min: 47.64, T-max: 1499   \n",
      "*** Episode 250 \t Average Score (over agents): 571.67 \t Average Score on 100 Episode: 535.34, Time: 01:16:16***\n",
      "Episode: 251, Score: 626.37, Max: 626.37, Min: 47.64, T-max: 1499   \n",
      "Episode: 252, Score: 608.43, Max: 626.37, Min: 47.64, T-max: 1499   \n",
      "Episode: 253, Score: 591.81, Max: 626.37, Min: 47.64, T-max: 1499   \n",
      "Episode: 254, Score: 596.37, Max: 626.37, Min: 47.64, T-max: 1499   \n",
      "Episode: 255, Score: 603.11, Max: 626.37, Min: 47.64, T-max: 1499   \n",
      "Episode: 256, Score: 601.63, Max: 626.37, Min: 47.64, T-max: 1499   \n",
      "Episode: 257, Score: 564.71, Max: 626.37, Min: 47.64, T-max: 1499   \n",
      "Episode: 258, Score: 538.93, Max: 626.37, Min: 47.64, T-max: 1499   \n",
      "Episode: 259, Score: 574.07, Max: 626.37, Min: 47.64, T-max: 1499   \n",
      "Episode: 260, Score: 597.06, Max: 626.37, Min: 47.64, T-max: 1499   \n",
      "*** Episode 260 \t Average Score (over agents): 597.06 \t Average Score on 100 Episode: 551.27, Time: 01:19:47***\n",
      "Episode: 261, Score: 602.12, Max: 626.37, Min: 47.64, T-max: 1499   \n",
      "Episode: 262, Score: 591.41, Max: 626.37, Min: 47.64, T-max: 1499   \n",
      "Episode: 263, Score: 623.06, Max: 626.37, Min: 47.64, T-max: 1499   \n",
      "Episode: 264, Score: 616.28, Max: 626.37, Min: 47.64, T-max: 1499   \n",
      "Episode: 265, Score: 621.68, Max: 626.37, Min: 47.64, T-max: 1499   \n",
      "Episode: 266, Score: 609.46, Max: 626.37, Min: 47.64, T-max: 1499   \n",
      "Episode: 267, Score: 637.57, Max: 637.57, Min: 47.64, T-max: 1499   \n",
      "Episode: 268, Score: 643.39, Max: 643.39, Min: 47.64, T-max: 1499   \n",
      "Episode: 269, Score: 606.98, Max: 643.39, Min: 47.64, T-max: 1499   \n",
      "Episode: 270, Score: 598.91, Max: 643.39, Min: 47.64, T-max: 1499   \n",
      "*** Episode 270 \t Average Score (over agents): 598.91 \t Average Score on 100 Episode: 563.47, Time: 01:23:09***\n",
      "Episode: 271, Score: 634.17, Max: 643.39, Min: 47.64, T-max: 1499   \n",
      "Episode: 272, Score: 639.34, Max: 643.39, Min: 47.64, T-max: 1499   \n",
      "Episode: 273, Score: 106.73, Max: 643.39, Min: 47.64, T-max: 1499   \n",
      "Episode: 274, Score: 171.04, Max: 643.39, Min: 47.64, T-max: 1499   \n",
      "Episode: 275, Score: 600.95, Max: 643.39, Min: 47.64, T-max: 1499   \n",
      "Episode: 276, Score: 616.35, Max: 643.39, Min: 47.64, T-max: 1499   \n",
      "Episode: 277, Score: -0.59, Max: 643.39, Min: -0.59, T-max: 7   \n",
      "Episode: 278, Score: -0.57, Max: 643.39, Min: -0.59, T-max: 7   \n",
      "Episode: 279, Score: -0.58, Max: 643.39, Min: -0.59, T-max: 7   \n",
      "Episode: 280, Score: -89.16, Max: 643.39, Min: -89.16, T-max: 1499   \n",
      "*** Episode 280 \t Average Score (over agents): -89.16 \t Average Score on 100 Episode: 539.57, Time: 01:25:35***\n",
      "Episode: 281, Score: -0.58, Max: 643.39, Min: -89.16, T-max: 7   \n",
      "Episode: 282, Score: -82.06, Max: 643.39, Min: -89.16, T-max: 1499   \n",
      "Episode: 283, Score: 472.67, Max: 643.39, Min: -89.16, T-max: 1499   \n",
      "Episode: 284, Score: 469.14, Max: 643.39, Min: -89.16, T-max: 1499   \n",
      "Episode: 285, Score: 491.03, Max: 643.39, Min: -89.16, T-max: 1499   \n",
      "Episode: 286, Score: 488.13, Max: 643.39, Min: -89.16, T-max: 1499   \n",
      "Episode: 287, Score: -0.60, Max: 643.39, Min: -89.16, T-max: 7   \n",
      "Episode: 288, Score: -111.56, Max: 643.39, Min: -111.56, T-max: 1499   \n",
      "Episode: 289, Score: -111.93, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 290, Score: 354.71, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "*** Episode 290 \t Average Score (over agents): 354.71 \t Average Score on 100 Episode: 506.70, Time: 01:28:25***\n",
      "Episode: 291, Score: 359.41, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 292, Score: 433.85, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 293, Score: 431.46, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 294, Score: 64.65, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 295, Score: 43.73, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 296, Score: 435.46, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 297, Score: 455.24, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 298, Score: 189.14, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 299, Score: -0.62, Max: 643.39, Min: -111.93, T-max: 7   \n",
      "Episode: 300, Score: 219.21, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "*** Episode 300 \t Average Score (over agents): 219.21 \t Average Score on 100 Episode: 479.87, Time: 01:31:34***\n",
      "Episode: 301, Score: 428.53, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 302, Score: 429.62, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 303, Score: 242.01, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 304, Score: 255.07, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 305, Score: 425.44, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 306, Score: 428.67, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 307, Score: 413.60, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 308, Score: 417.67, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 309, Score: 446.84, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 310, Score: 449.82, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "*** Episode 310 \t Average Score (over agents): 449.82 \t Average Score on 100 Episode: 462.26, Time: 01:35:00***\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 311, Score: 249.82, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 312, Score: 276.07, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 313, Score: 465.39, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 314, Score: 465.62, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 315, Score: 401.44, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 316, Score: 403.94, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 317, Score: 495.87, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 318, Score: 497.52, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 319, Score: 390.50, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 320, Score: 414.67, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "*** Episode 320 \t Average Score (over agents): 414.67 \t Average Score on 100 Episode: 444.65, Time: 01:38:26***\n",
      "Episode: 321, Score: 502.47, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 322, Score: 506.11, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 323, Score: 494.22, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 324, Score: 502.56, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 325, Score: 517.22, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 326, Score: 506.04, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 327, Score: 480.22, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 328, Score: 472.92, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 329, Score: 507.33, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 330, Score: 513.19, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "*** Episode 330 \t Average Score (over agents): 513.19 \t Average Score on 100 Episode: 434.88, Time: 01:42:16***\n",
      "Episode: 331, Score: 486.63, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 332, Score: 500.28, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 333, Score: 344.83, Max: 643.39, Min: -111.93, T-max: 1000   \n",
      "Episode: 334, Score: 511.93, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 335, Score: 524.42, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 336, Score: 230.09, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 337, Score: 162.91, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 338, Score: 522.52, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 339, Score: 531.94, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 340, Score: 17.29, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "*** Episode 340 \t Average Score (over agents): 17.29 \t Average Score on 100 Episode: 413.03, Time: 01:45:24***\n",
      "Episode: 341, Score: -10.19, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 342, Score: 515.65, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 343, Score: 504.87, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 344, Score: 484.26, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 345, Score: 483.82, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 346, Score: 503.54, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 347, Score: 490.98, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 348, Score: 472.82, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 349, Score: 474.22, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 350, Score: 511.76, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "*** Episode 350 \t Average Score (over agents): 511.76 \t Average Score on 100 Episode: 405.96, Time: 01:48:40***\n",
      "Episode: 351, Score: 509.36, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 352, Score: 186.70, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 353, Score: 189.99, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 354, Score: 485.59, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 355, Score: 490.28, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 356, Score: 480.60, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 357, Score: 473.11, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 358, Score: 475.17, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 359, Score: 471.71, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 360, Score: 493.74, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "*** Episode 360 \t Average Score (over agents): 493.74 \t Average Score on 100 Episode: 389.50, Time: 01:51:57***\n",
      "Episode: 361, Score: 501.51, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 362, Score: 491.94, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 363, Score: 478.65, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 364, Score: 526.31, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 365, Score: 524.20, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 366, Score: 477.37, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 367, Score: 472.57, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 368, Score: 475.99, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 369, Score: 522.32, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 370, Score: 497.69, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "*** Episode 370 \t Average Score (over agents): 497.69 \t Average Score on 100 Episode: 377.68, Time: 01:55:13***\n",
      "Episode: 371, Score: 306.98, Max: 643.39, Min: -111.93, T-max: 1000   \n",
      "Episode: 372, Score: 332.96, Max: 643.39, Min: -111.93, T-max: 1000   \n",
      "Episode: 373, Score: 503.72, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 374, Score: 519.42, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 375, Score: 503.61, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 376, Score: 527.35, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 377, Score: 535.05, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 378, Score: 551.77, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 379, Score: 507.54, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 380, Score: 535.38, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "*** Episode 380 \t Average Score (over agents): 535.38 \t Average Score on 100 Episode: 399.14, Time: 01:58:18***\n",
      "Episode: 381, Score: 531.57, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 382, Score: 517.14, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 383, Score: 545.33, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 384, Score: 515.39, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 385, Score: 566.73, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 386, Score: 566.05, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 387, Score: 511.43, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 388, Score: 421.89, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 389, Score: 529.01, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 390, Score: 555.09, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "*** Episode 390 \t Average Score (over agents): 555.09 \t Average Score on 100 Episode: 432.05, Time: 02:01:34***\n",
      "Episode: 391, Score: 541.19, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 392, Score: 533.28, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 393, Score: 556.68, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 394, Score: 559.23, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 395, Score: 568.92, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 396, Score: 556.06, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 397, Score: 583.61, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 398, Score: 547.54, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 399, Score: 561.45, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 400, Score: 381.65, Max: 643.39, Min: -111.93, T-max: 1000   \n",
      "*** Episode 400 \t Average Score (over agents): 381.65 \t Average Score on 100 Episode: 459.63, Time: 02:04:41***\n",
      "Episode: 401, Score: 569.25, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 402, Score: 581.92, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 403, Score: 582.27, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 404, Score: 584.98, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 405, Score: 573.23, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 406, Score: 569.73, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 407, Score: 571.60, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 408, Score: 566.85, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 409, Score: 574.08, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 410, Score: 558.20, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "*** Episode 410 \t Average Score (over agents): 558.20 \t Average Score on 100 Episode: 477.58, Time: 02:07:59***\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 411, Score: 560.86, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 412, Score: 564.75, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 413, Score: 566.51, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 414, Score: 578.47, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 415, Score: 583.40, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 416, Score: 576.54, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 417, Score: 560.69, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 418, Score: 579.91, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 419, Score: 579.88, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 420, Score: 584.64, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "*** Episode 420 \t Average Score (over agents): 584.64 \t Average Score on 100 Episode: 494.32, Time: 02:11:15***\n",
      "Episode: 421, Score: 557.40, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 422, Score: 558.32, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 423, Score: 564.74, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 424, Score: 588.60, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 425, Score: 572.83, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 426, Score: 587.47, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 427, Score: 595.37, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 428, Score: 594.11, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 429, Score: 607.11, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 430, Score: 578.81, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "*** Episode 430 \t Average Score (over agents): 578.81 \t Average Score on 100 Episode: 502.35, Time: 02:14:31***\n",
      "Episode: 431, Score: 585.94, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 432, Score: 566.74, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 433, Score: 581.97, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 434, Score: 597.20, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 435, Score: 587.69, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 436, Score: 606.55, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 437, Score: 588.40, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 438, Score: 582.19, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 439, Score: 580.69, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 440, Score: 610.89, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "*** Episode 440 \t Average Score (over agents): 610.89 \t Average Score on 100 Episode: 522.90, Time: 02:17:47***\n",
      "Episode: 441, Score: 600.31, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 442, Score: 590.55, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 443, Score: 583.58, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 444, Score: 596.53, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 445, Score: 615.25, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 446, Score: 602.85, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 447, Score: 590.94, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 448, Score: 623.32, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 449, Score: 615.65, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 450, Score: 609.12, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "*** Episode 450 \t Average Score (over agents): 609.12 \t Average Score on 100 Episode: 538.87, Time: 02:21:03***\n",
      "Episode: 451, Score: 598.06, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 452, Score: 616.82, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 453, Score: 624.81, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 454, Score: 620.46, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 455, Score: 581.79, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 456, Score: 625.38, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 457, Score: 589.27, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 458, Score: 640.99, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 459, Score: 637.58, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 460, Score: 607.58, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "*** Episode 460 \t Average Score (over agents): 607.58 \t Average Score on 100 Episode: 557.73, Time: 02:24:19***\n",
      "Episode: 461, Score: 589.24, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 462, Score: 640.87, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 463, Score: 637.32, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 464, Score: 487.70, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 465, Score: 508.04, Max: 643.39, Min: -111.93, T-max: 1499   \n",
      "Episode: 466, Score: 646.84, Max: 646.84, Min: -111.93, T-max: 1499   \n",
      "Episode: 467, Score: 633.61, Max: 646.84, Min: -111.93, T-max: 1499   \n",
      "Episode: 468, Score: 635.46, Max: 646.84, Min: -111.93, T-max: 1499   \n",
      "Episode: 469, Score: 646.28, Max: 646.84, Min: -111.93, T-max: 1499   \n",
      "Episode: 470, Score: 614.10, Max: 646.84, Min: -111.93, T-max: 1499   \n",
      "*** Episode 470 \t Average Score (over agents): 614.10 \t Average Score on 100 Episode: 568.44, Time: 02:27:36***\n",
      "Episode: 471, Score: 618.30, Max: 646.84, Min: -111.93, T-max: 1499   \n",
      "Episode: 472, Score: 635.07, Max: 646.84, Min: -111.93, T-max: 1499   \n",
      "Episode: 473, Score: 444.08, Max: 646.84, Min: -111.93, T-max: 1000   \n",
      "Episode: 474, Score: 664.21, Max: 664.21, Min: -111.93, T-max: 1499   \n",
      "Episode: 475, Score: 659.77, Max: 664.21, Min: -111.93, T-max: 1499   \n",
      "Episode: 476, Score: 660.75, Max: 664.21, Min: -111.93, T-max: 1499   \n",
      "Episode: 477, Score: 628.53, Max: 664.21, Min: -111.93, T-max: 1499   \n",
      "Episode: 478, Score: 622.08, Max: 664.21, Min: -111.93, T-max: 1499   \n",
      "Episode: 479, Score: 653.74, Max: 664.21, Min: -111.93, T-max: 1499   \n",
      "Episode: 480, Score: 661.61, Max: 664.21, Min: -111.93, T-max: 1499   \n",
      "*** Episode 480 \t Average Score (over agents): 661.61 \t Average Score on 100 Episode: 582.68, Time: 02:30:47***\n",
      "Episode: 481, Score: 660.25, Max: 664.21, Min: -111.93, T-max: 1499   \n",
      "Episode: 482, Score: 424.61, Max: 664.21, Min: -111.93, T-max: 1000   \n",
      "Episode: 483, Score: 648.43, Max: 664.21, Min: -111.93, T-max: 1499   \n",
      "Episode: 484, Score: 438.50, Max: 664.21, Min: -111.93, T-max: 1000   \n",
      "Episode: 485, Score: 636.05, Max: 664.21, Min: -111.93, T-max: 1499   \n",
      "Episode: 486, Score: 645.00, Max: 664.21, Min: -111.93, T-max: 1499   \n",
      "Episode: 487, Score: 656.30, Max: 664.21, Min: -111.93, T-max: 1499   \n",
      "Episode: 488, Score: 649.07, Max: 664.21, Min: -111.93, T-max: 1499   \n",
      "Episode: 489, Score: 644.71, Max: 664.21, Min: -111.93, T-max: 1499   \n",
      "Episode: 490, Score: 625.88, Max: 664.21, Min: -111.93, T-max: 1499   \n",
      "*** Episode 490 \t Average Score (over agents): 625.88 \t Average Score on 100 Episode: 590.38, Time: 02:33:49***\n",
      "Episode: 491, Score: 460.89, Max: 664.21, Min: -111.93, T-max: 1000   \n",
      "Episode: 492, Score: 681.69, Max: 681.69, Min: -111.93, T-max: 1499   \n",
      "Episode: 493, Score: 663.53, Max: 681.69, Min: -111.93, T-max: 1499   \n",
      "Episode: 494, Score: 677.99, Max: 681.69, Min: -111.93, T-max: 1499   \n",
      "Episode: 495, Score: 660.52, Max: 681.69, Min: -111.93, T-max: 1499   \n",
      "Episode: 496, Score: 674.12, Max: 681.69, Min: -111.93, T-max: 1499   \n",
      "Episode: 497, Score: 668.70, Max: 681.69, Min: -111.93, T-max: 1499   \n",
      "Episode: 498, Score: 667.27, Max: 681.69, Min: -111.93, T-max: 1499   \n",
      "Episode: 499, Score: 650.78, Max: 681.69, Min: -111.93, T-max: 1499   \n",
      "Episode: 500, Score: 673.20, Max: 681.69, Min: -111.93, T-max: 1499   \n",
      "*** Episode 500 \t Average Score (over agents): 673.20 \t Average Score on 100 Episode: 601.27, Time: 02:37:01***\n",
      "Episode: 501, Score: 683.91, Max: 683.91, Min: -111.93, T-max: 1499   \n",
      "Episode: 502, Score: 678.37, Max: 683.91, Min: -111.93, T-max: 1499   \n",
      "Episode: 503, Score: 676.12, Max: 683.91, Min: -111.93, T-max: 1499   \n",
      "Episode: 504, Score: 673.51, Max: 683.91, Min: -111.93, T-max: 1499   \n",
      "Episode: 505, Score: 683.12, Max: 683.91, Min: -111.93, T-max: 1499   \n",
      "Episode: 506, Score: 681.77, Max: 683.91, Min: -111.93, T-max: 1499   \n",
      "Episode: 507, Score: 699.07, Max: 699.07, Min: -111.93, T-max: 1499   \n",
      "Episode: 508, Score: 703.57, Max: 703.57, Min: -111.93, T-max: 1499   \n",
      "Episode: 509, Score: 690.54, Max: 703.57, Min: -111.93, T-max: 1499   \n",
      "Episode: 510, Score: 711.25, Max: 711.25, Min: -111.93, T-max: 1499   \n",
      "*** Episode 510 \t Average Score (over agents): 711.25 \t Average Score on 100 Episode: 612.76, Time: 02:40:17***\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 511, Score: 471.96, Max: 711.25, Min: -111.93, T-max: 1000   \n",
      "Episode: 512, Score: 695.10, Max: 711.25, Min: -111.93, T-max: 1499   \n",
      "Episode: 513, Score: 680.31, Max: 711.25, Min: -111.93, T-max: 1499   \n",
      "Episode: 514, Score: 692.46, Max: 711.25, Min: -111.93, T-max: 1499   \n",
      "Episode: 515, Score: 662.88, Max: 711.25, Min: -111.93, T-max: 1499   \n",
      "Episode: 516, Score: 678.53, Max: 711.25, Min: -111.93, T-max: 1499   \n",
      "Episode: 517, Score: 652.39, Max: 711.25, Min: -111.93, T-max: 1499   \n",
      "Episode: 518, Score: 652.23, Max: 711.25, Min: -111.93, T-max: 1499   \n",
      "Episode: 519, Score: 665.88, Max: 711.25, Min: -111.93, T-max: 1499   \n",
      "Episode: 520, Score: 657.87, Max: 711.25, Min: -111.93, T-max: 1499   \n",
      "*** Episode 520 \t Average Score (over agents): 657.87 \t Average Score on 100 Episode: 620.50, Time: 02:43:28***\n",
      "Episode: 521, Score: 668.87, Max: 711.25, Min: -111.93, T-max: 1499   \n",
      "Episode: 522, Score: 692.43, Max: 711.25, Min: -111.93, T-max: 1499   \n",
      "Episode: 523, Score: 665.90, Max: 711.25, Min: -111.93, T-max: 1499   \n",
      "Episode: 524, Score: 664.24, Max: 711.25, Min: -111.93, T-max: 1499   \n",
      "Episode: 525, Score: 675.67, Max: 711.25, Min: -111.93, T-max: 1499   \n",
      "Episode: 526, Score: 668.25, Max: 711.25, Min: -111.93, T-max: 1499   \n",
      "Episode: 527, Score: 679.82, Max: 711.25, Min: -111.93, T-max: 1499   \n",
      "Episode: 528, Score: 672.25, Max: 711.25, Min: -111.93, T-max: 1499   \n",
      "Episode: 529, Score: 680.64, Max: 711.25, Min: -111.93, T-max: 1499   \n",
      "Episode: 530, Score: 678.00, Max: 711.25, Min: -111.93, T-max: 1499   \n",
      "*** Episode 530 \t Average Score (over agents): 678.00 \t Average Score on 100 Episode: 629.91, Time: 02:46:44***\n",
      "Episode: 531, Score: 693.37, Max: 711.25, Min: -111.93, T-max: 1499   \n",
      "Episode: 532, Score: 688.02, Max: 711.25, Min: -111.93, T-max: 1499   \n",
      "Episode: 533, Score: 681.24, Max: 711.25, Min: -111.93, T-max: 1499   \n",
      "Episode: 534, Score: 679.73, Max: 711.25, Min: -111.93, T-max: 1499   \n",
      "Episode: 535, Score: 684.77, Max: 711.25, Min: -111.93, T-max: 1499   \n",
      "Episode: 536, Score: 686.82, Max: 711.25, Min: -111.93, T-max: 1499   \n",
      "Episode: 537, Score: 693.68, Max: 711.25, Min: -111.93, T-max: 1499   \n",
      "Episode: 538, Score: 691.17, Max: 711.25, Min: -111.93, T-max: 1499   \n",
      "Episode: 539, Score: 713.98, Max: 713.98, Min: -111.93, T-max: 1499   \n",
      "Episode: 540, Score: 691.24, Max: 713.98, Min: -111.93, T-max: 1499   \n",
      "*** Episode 540 \t Average Score (over agents): 691.24 \t Average Score on 100 Episode: 640.07, Time: 02:50:00***\n",
      "Episode: 541, Score: 717.27, Max: 717.27, Min: -111.93, T-max: 1499   \n",
      "Episode: 542, Score: 714.99, Max: 717.27, Min: -111.93, T-max: 1499   \n",
      "Episode: 543, Score: 720.76, Max: 720.76, Min: -111.93, T-max: 1499   \n",
      "Episode: 544, Score: 709.81, Max: 720.76, Min: -111.93, T-max: 1499   \n",
      "Episode: 545, Score: 700.99, Max: 720.76, Min: -111.93, T-max: 1499   \n",
      "Episode: 546, Score: 719.85, Max: 720.76, Min: -111.93, T-max: 1499   \n",
      "Episode: 547, Score: 739.27, Max: 739.27, Min: -111.93, T-max: 1499   \n",
      "Episode: 548, Score: 726.52, Max: 739.27, Min: -111.93, T-max: 1499   \n",
      "Episode: 549, Score: 731.94, Max: 739.27, Min: -111.93, T-max: 1499   \n",
      "\n",
      " End of PPO!\n"
     ]
    }
   ],
   "source": [
    "def ppo(env, agent, n_episodes=2000, print_every=10, n_agent = 12):\n",
    "    \"\"\"\n",
    "    Params\n",
    "    ======\n",
    "        n_episodes (int): maximum number of training episodes\n",
    "        max_t (int): maximum number of timesteps per episode\n",
    "        print_every (int): number of episodes to print result\n",
    "        n_agent (int): number of identical agents in environment\n",
    "    \"\"\"\n",
    "    scores_deque = deque(maxlen=100)\n",
    "    scores_global = []    \n",
    "    scores = []\n",
    "    max_t = 1500\n",
    "        \n",
    "    time_start = time.time()\n",
    "    \n",
    "    for i_episode in range(1, n_episodes+1):\n",
    "        env_info = env.reset(train_mode=True)[brain_name]\n",
    "        states = env_info.vector_observations\n",
    "        agent_scores = np.zeros(n_agent)\n",
    "        t_max = 0\n",
    "        for t in range(max_t):\n",
    "            actions, log_probs, _, values = agent.act(states)\n",
    "            # get needed information from environment\n",
    "            env_info = env.step(actions)[brain_name]\n",
    "            next_states = env_info.vector_observations\n",
    "            rewards = env_info.rewards\n",
    "            dones = np.array([1 if t else 0 for t in env_info.local_done])\n",
    "            agent.save_step([states, values.detach(), actions, log_probs.detach(), rewards, 1 - dones])\n",
    "            states = next_states\n",
    "            agent_scores += rewards\n",
    "            t_max = t\n",
    "            if all(dones) or t == max_t-1:\n",
    "                agent.step(next_states)\n",
    "                break\n",
    "                \n",
    "        score = np.mean(agent_scores)\n",
    "        scores_deque.append(score)\n",
    "        scores.append(score)\n",
    "\n",
    "        print('Episode: {}, Score: {:.2f}, Max: {:.2f}, Min: {:.2f}, T-max: {}   '\\\n",
    "              .format(i_episode, score, np.max(scores), np.min(scores), t_max))\n",
    "        \n",
    "        if i_episode % print_every == 0:\n",
    "            torch.save(agent.actor_critic.actor.state_dict(), 'checkpoint_actor.pth')\n",
    "            torch.save(agent.actor_critic.critic.state_dict(), 'checkpoint_critic.pth')\n",
    "            s = (int)(time.time() - time_start) \n",
    "            print('*** Episode {} \\t Average Score (over agents): {:.2f} \\t Average Score on 100 Episode: {:.2f}, Time: {:02}:{:02}:{:02}***'\\\n",
    "                  .format(i_episode, score, np.mean(scores_deque), s//3600, s%3600//60, s%60))\n",
    "            \n",
    "        if len(scores_deque) == 100 and np.mean(scores_deque) >= 650: # 500:   \n",
    "            break\n",
    "\n",
    "\n",
    "    print('\\n End of PPO!')\n",
    "    return scores\n",
    "\n",
    "scores = ppo(env=env, agent=agent)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJztnXecXGW5+L/vbG/Zmr5JNj0EQhJIAiF0kK5wQQWuFxBRLIjlXq+iF+Riu6hX8eIPKQqCKMWCgIBAEnoNIbT03nvZ3WzfmXl/f5wy55w5Mzuzu7M75fnmk8+c8p6Z9+zsvs95utJaIwiCIAheAoM9AUEQBCE9EQEhCIIg+CICQhAEQfBFBIQgCILgiwgIQRAEwRcREIIgCIIvIiAEQRAEX0RACIIgCL6IgBAEQRB8yR/sCfSFuro63dDQMNjTEARByCjefffd/VrroT2Ny2gB0dDQwNKlSwd7GoIgCBmFUmpLIuPExCQIgiD4IgJCEARB8EUEhCAIguCLCAhBEATBFxEQgiAIgi8iIARBEARfREAIgiAIvoiAEARByBDCYc0jS7bSGQwNyOeJgBAEQcgQnluxmxse+4jbF68bkM8TASEIgpAhNLV3A3DHixt46sOdKf88ERCCIAgZwu7mDnt7xJDilH9eRtdiEgRByAXCYc2+lk62H2oH4MwjhnHsuOqUf64ICEEQhDRm3+FOrr5/Cct3NDN5WDnHjqvmd1fNHZDPFhOTIAhCmrJk00HO+MVLLN/RDMC6vS3UV5cM2OeLBiEIgpAiLrvnTcqLCvjdVXN6df2n734z6pgICEEQhAznYGsXb2082C/vNWtMFWVFeby+/gD11aX98p6JICYmQRCEfqa1M8gxP1zYr+85r6EWGFgNQgSEIAhCP/PkB+4chf0tnUm/R3cobG+Hteb8o0cyt6GaGaMr+zy/RBEBIQiC0M+s29Pi2p/zo0X84c3N9r7WmjteXM/upkheQ2cwxFX3LWHpZsMs9c1H37fPBUOaScPK+cuXTqCqtDClc3ciAkIQBKGf2XqwNerY959YQSisAViz5zA/f26NSwis3d3Cy2v38cm73uRQaxdPfbjLPnfxMaNTP2kfxEktCIKQJMFQGA0U5EWesZ/5aBdhrZnbUMOiVXs584hhLN/R7Mp+bmzrora8iOb2IBApnQGwcX9E67Cilx76/HHMGltFSUFeiu/IHxEQgiAISXLKz19id3MHb9xwOsPNkhdf+dMyAM6bMQKAqSMqAOUSEM8s380Vx49j72HjWCAAew93cNPjy3luxR573Lq9Lfzn2VM5YVLdAN2RP2JiEgRBSJIdje2EwprjfrKYD7Y1upzQz3y0m+FDivjSKRPJ86ywNz2+nL3NHexpNsZv2NvKvB8vtoVDXkDxk3+ZwTlHjuBzC8YP2P3EQgSEIAhCH7jwjtd5dvluY3vWKACuP30yFcUF5AUUgP0KsL+li72mVtHe7e7r8Pb3zuBfjxvLXVccS0nh4JiVnIiJSRAEwURrTVi7F/REuPHx5QD813lH8K2zpjK6yshVUMp4n7ryQltreGntXhau3BP1Hg994Tjqyov6Mv1+RzQIQRAEk68+9B7Tbvone5o7aGrv5q/vbqe9K/HubbXlRYypKSVgaQ62gIgs/D97dg0b90einCYPK2fRv5/CCRMH19/ghwgIQRAEk6c/2kV3SHPp3W9yzysb+NZfPuCXC9cA0NEdsvMWpg6vsK+Z2xApu+3VPKzd4oI8rl7Q4Dp3xfHj+OM1x/HnL85n0rDyFNxN3xEBIQiCYDJzTBUAmw+0sWjlXgDbNPS9xz7i+P9ZTGtnEKXgY9OHs/g/TuHuK2IX4nNqEjd//EjXuasXNHDi5DqqywYu8S1ZREAIgiD4sGbPYQA27Gth474WHntvBwAvrN5LWGvylGLi0HJq4izwAdPEFDBX2jE1hm9i4TdPZsLQ9NQanIiTWhD6gW0H2xhdVWI/MQrpz/ZDbZz40xc5/+iRNLV188fPH0fYzHR2smJnM6f/4mWK8gN0BsPc88pGgmHtMif9/uq5vi1ALR+EJSieuv4kDrV20VBXlqK76l9EQAhCH3l57T6uum8Jt18+m6NHV1JbXkhFccFgT0vw8Mrafaza1cy5R43kkXe22i07nzZLWnQGQ4R1tICw6AyGOXZcNe9uOURBnuLIUZGieadNHeZ7jaU5WMKksqSAypLM+d0QASEIfeS+1zYBsHJnM197+D0umjWKX102e5BnJXi56YnlbDnQxjPLd/PBtkaOn1DjOr9iZzMHW7soKciLyk+wuOqEBt7dcojukCYvAWUx4NEgMg3xQQhCH9l72HBi/vXdbQAcbOuON1wYJEZVGvb/tbsN34K34urFv3mDXU0d1Jb7+xTqq0uYWR/RGhJZ9C3NIVMtjyIgBCEOzR3d7GxsjzumtdMovLa/pQuACRliX841LPORpR1Y+86QVTByGexthwO6MD/AyMoSe7FPxN9kCZFkE+/SBREQQtbT0hnscZEH6AoaDVrCYc06M4Ll03e9yQm3vuA7/rkVu/lweyM7EnhvYeAIhsLc+dIGmju6eX39fi749assuPUF3t7kbv/Z0W1831fMH+c6Proq4mx2ahN5SlGYH2BYRbG93xOZbmISH4SQ9Zx92yvsaGxn863nxxyzu6mD4/9nMbdePIMlmw7y2Hs7WHrjmaw2zRGtnUHKitx/Ll988F3f94rn6BRSzzubD/HTZ1fzwbZGDrV1sXxHs++4zqChSXjLWxTkBThpch2vrttPueM7t7QAq0ZSYhqE+9pMQwSEkNV0dIfsJ/yQJzTRySaz9MENj33kutZiy4E2po8aYu/HK7+gNXz1oWVo4I5/PaYv0xeS5IE3NrN+r+FbeHbF7rgRQ1ZE69AKt88hTynu++xcmtu7+dIfIw8BEW3AHJeA/SXig8hMAZFSE5NSqkop9Vel1Gql1Cql1HylVI1SaqFSap35Wm2OVUqp25VS65VSHyql5C9L6DMb90Vq3hxsNXwE9762iV8+v4bWziDbDrbR2hnkQGt0z2CtYUix8Qzl7RB2sK0r5mdqNE99uMsOnxRSy6vr9vHAG5sBuPnJFTz41hb7nLMhTyzKiwoYPiSiRQQCioK8ALXlRXaxPYgs9vlm7Goii76yE+UyU0CkWoP4P+BZrfUnlVKFQCnwPWCx1vpWpdQNwA3Ad4Bzgcnm/+OAO81XQeg1zR2RBeKaB97hqvkN/PCplQDc/sJ6AC6ePZpxtdGO5bDW1FUU0dwRZPOBNte5Q6awuWzuGB55ZxsTh5axwRRG3UExMaWC+17bhAauOdHok/DQ21upry7hyvuWAPDxmaN69b55AcWbN5zBv/zmdT7Y3uTyLTi3A57S3YlFMVnv06upDTop0yCUUkOAk4F7AbTWXVrrRuBC4AFz2APAReb2hcAftMFbQJVSamSq5ifkBs2OJ8gPtzfxH3/5IGrMY+/t4LZFa6OOaw355mLw+Hs7uOiO13l3y0FCYc1NTxjlnS8+pp6VPzibi2ZFegZv8elHLPSdHzy10hbuAN/7+0e2cACj5aeF84F9yvBy3+MW+QFFIKAiC79jVXSaJPM8/oRE/AqBDNcgUmlimgDsA36vlHpPKfU7pVQZMFxrvQvAfLVSEEcD2xzXbzePCUKPdAXDPPDGZoIhIzLlYGsXt/xjBU+8vzPmNSdPGcq0ERUxz4e1prXT8DWs3n2Y97c1cudLG3huxW7e29oIQHVpAaWF+Zw8Zah9nUQ19T9eU9Hf39tub1uhqI+btZIAqkojfoWxNaX29kgzF8JJfl5szcCpJHgFQzICQiECwks+cAxwp9Z6NtCKYU6Khd9PMEpXV0pdq5RaqpRaum/fvv6ZqZDx3P3yBm5+cgWPLTMWiX8u38XvX9/M0x/5+wHG1JTwy0/PZJZZvdOPsI7u+NXcHmSRo9mLtRAdXV/Jf398OgChkJiY+puTfhoJNe7oDvHNRyOaoPXTXrrlkH2sqrSAhz5/HFfNH+cqezKiMrpekten4NIafLbzkzIxmQIiM+VDSgXEdmC71vptc/+vGAJjj2U6Ml/3OsaPcVxfD0Q9/mmt79Faz9Fazxk6dKj3tJCjbD9kPLU/v3IPM25+jt+9usl1/qzpwwH47AkNANx8wZHUlRdRYTqhx9eV8bUzJnveVdPWFbT3po8cQlN7N7uaIk3oq0qNxUcpxWcXjGf4kCJCEubab+xobOfulzfQ3BH5Hj7c3uQaYwUfgNG5bWZ9JXXlRZwwqY5bLjzKtTj7FdSLp0EEfJzUyWRHW2MyVD6kzkmttd6tlNqmlJqqtV4DnAGsNP9fBdxqvj5hXvIk8FWl1CMYzukmyxQlCD3Rai7ki1YZT/eHOyMLSlF+gLv+7Vga27spL8rn6PpKTptmWDbH1xn26f+5eAb5AcXti9fZ1wXDmo7uMF85dSJXzB/HL55fy+vr99MdCnP6tGF855xpFHhiHRWKoE9F0FymuaOb7Qfb7TDhZ5fvYmdjB58znc0WrZ1BdjS2M2V4BS+s3oNSilueXGEHCHzzzCnctmgtn777zajPmNdQw5LNB2lq7+ahLxyPU0Y7F/nvnjeNovwAz67YTZsZqpwfx3TkJyySMjFlqO/BItVRTNcDfzIjmDYCV2NoLX9WSl0DbAU+ZY59BjgPWA+0mWMFIYo9zR0U5QcoLsjjyvuWcN1pk+xyF350BsMEAsqu23/xMfX2uUvnjuHESXWMrS1ljZkUZ2EtIJUlBYysLKGypICm9m6a27s5deowpvr4L5Qy+hoLET573xKWbW1k40/Oc4WhXjl/HPkOAXv1/e+wZNNBfnbJ0Xz7bx8CbtPM0WMqXe/7yn+exsk/fxGAi48ZzZLNB7lqfgNTvKUzzO/99stnU19dyi8vnUXnQ8vsMOR4mkGej8M6mSimTM1/sEipgNBavw/4tVs6w2esBq5L5XyEzKetK8hxP1nMzPpKrl4wniWbDjJ95F7bmewkP9Dz03xeQDG21nBiWuYmC0volJqZs5UlBbbQcMbNO1EgJiYPy0yH/uHOIAsd/pv1+1oYV1NGfp6itTPIErMUhiUcAMoL821t0GkeKi3MY2xtKY9eezx3vryBk6cMZeUPzqYoPy/q879x5hSGDSnm/BmRoMh8hxSwtMA8n4ijQIyQV0hMg0ikHEc6I5nUQkbx7PLdAHywvYk/LzWC3tbvbXHlO8xtqOadzYfICyi+dsbkhO2/QzxZt5aAKCk0/kyGOATIcB9bNhi+CL+mMwJs3t/K7uYOLp83hoeXbOOLD77LlgNtfGz6cK44flzU+ML8ACWFeRzuDFKQpzhi5BC+d940fvLMaooLDEFw3IRajptQG/dzSwrz7NwJCz/ns7M9qIVLWNgO58R9ENZbZaqckGJ9QkZhOaMB3thwAKXgtfX77ZpJAL++3EjCXzCpjq+dMZnro5zP/pR7ai1ZvYgtE0VlaUSAxOsjnIvyQWvNUx/uZOM+dwntO15cb2+/s9nQEM6bMZJ5DTVsMX0LC1fuYdlWIwKpxFz4pw6voCsYZu/hTvICir9/ZQEAJ0ysAwy/Ul9wahD5HsEQyweRZ4esmucS0SAy3AchAkLIKJwRRAD/dlz0k2dteSGvfec0fn1535r2WEJn0jDDke2s6xOrxk8gYNR8yjXW7mnhqw+9x+m/eNl1/OfPrbG3399mmJqqSgr57VVzXFrDnS9t4MhRQ6gz6yLNdPgbvn32VI4abeyPM82B0RFnyZFnhrbmBZStEeR5NARwZ0B7Q1aTqeaaqYiAEDKKPc0drsSnmy6YzgkT3SaGgrwA9dWlUdVXE2GUI05+9e5mSgvzGF1lJFfVlEX8DrEEhELlZDVXv5pH3hLrTzmcwpUlBfzwoqP4/gVG7khnMMxvr5zDDFMQWAIB4HiHCamiuIDNt57P5fPG9mm+3sglwFYNejQxmfsSxSQIaURzRzcvrN7LaVOHUl6UT3t3iML8AL+6bBZ/Wbrd9bTaWx6/bgG3LVrLw0u2seVAGw21ZfYf+dCKiICoiiUgVG6W+2515Its3NfCfa9vYm9zdAFEcC+scxtqKC4I8KtLZzGqqoSffXImF84abUciNdSWcnR9pe/79AVrDgWOuUQW/si4gI82EfFBiJNaENKG372yEYBRVSXcfUUkOG5YRTHXnTapXwTEsCHFnH3kCB5eso3uYJiigshq4ewu5nVoWyhywwfR1hVk7o8W8ctLZ3H2kSNoc0SR3fj4ct7YcAAwEhQXTKrj5idX2OedC/CM+kpW3nKOLYTLi/I5+8gRALz67dMYUVnsMvn0F74ahInzqd9Pm7B9EEkkymUqYmISMoaVu4zGL18/YzKF+QEKPY7Kn14ygxvPP6LPn2M9GQbD2hUBZUXOQGzzglIqJ3wQa/e00NoV4vbF6+gOhXnkna32OWdo8YjKYq70dGzzPnnHMsOMqSmNSkTsL6z8C7/3dwuFyHGrnpLtg8j01T8BRIMQ0oLGti7KivJ9/2BX7GyiozvE+r0tnDdjBMNihJheOrdvdmkLawEIaZ20kzH7lwyDXaZ/oba8iHte2cir6/bb55w/g+FDDA3g/Bkj7bpY+YHBfy710yD8TUfO8+5jyfgXMrVYnwgIIS049keLmDysnH9+/ST2tXSycV8ro6tKWLRqD7f8wyjxHFDwiV7W/E8Ga4EIh3XS8esZbnJOGKv8RXF+gG0H3b0ydjZFnNNWvsjZR42wBUQayIeowntO3At/bG0wEf9CpuuSIiCEQaelM0gorFm9+zD//ucP+LujbLOTsIZ6RwRTqnBqEF779z++eiJdodjtRlNhL+8rOxrbCYc1Y/rxZ7durxEC/LwjM9pi28GIgCgzs9CdC3E6mGbyfUJabSd1D9Ozpp/pEUqJIAJCGDSaO7oJhTR7D0eiXWIJBwu/apz9jWUO0DraZDSjh4gal3EiTdaPBbcapbI333p+UtdZGpRX6IXCmlfWxi+1f9LkOgJK2YltflVRB5O8OFKgp/lZt5LpOQ6JkAbKnpCrnP6/LzP7hwvZfsgwUcwZV93jNbFKXPQnzvUhaR9EdFj9oPHIkq28u+Vgr6+/6vdLOPf/XqU7FKYzGLKLEK7dc5j9LbF7cgMcOaqSBz43z84+z4sRGTRY+JmWLHrSAq0HiGT852lwy71CBIQwaOxvMTQHy559iqMrW311dOcvGBgNIuByXCZ3rXI5NQdvVWjtDHLDYx9xyZ3RpbET5dV1RgmTNzYcYOqNz3LHi+vpDoWjem34Ueh5Qk83E1OejyMk2eikZB4eMjU1RkxMQkrZfqiNO1/awM0fP5I/vrWFj3Y0sf1Qm6vJy4NvbqYwP8CxDRENYtqIClfdJYshJan/lXX+2fdFgxgonnh/B/e8spF/fPVEW7itMkOCnfN68M3NnHXkiKS1MEvDu/+NzeQFAvxt2fYerogOH/XLSB5M4mkQPWk4EuYqCP3ET55ZxTMf7eaUKUP5gaPhvJPNB9oYX1dm92sA+M+zp/HlUydxyZ1vAHDPFceyfGfzgDyVuxyXffi4gVo+vv7I+wBs3N/CT55ZTXVpoV3LqCBP0R3SaA03PbGCu17eyOs3nA7A6+v3s2zLIa47bVLUou3sabHP9BG1dobYerA16vNv/vh07n55I7XlhazYaQimAk+Oil9xvMHErxWonQCXoA8iGTLVxCQCQkgpVoXUd7ceijuuMC9AVUlEQIyoLHbVOzrryBGcZWbYppq+rF8D6bjUWvPmxgMU5gXoCoV5dvluXlhtdPAdVWVoCXXlRa4Chzsc9ZF+/twa3t/WyJiaUi6aPdr13h3dYXt7T7NxfXt3yO6H4eSyuWO5esF4mtq6mfmD54FoIRCrjedgEU9IOU/5mYacQQw9kammJQsREEJKqSs36hfd/fJG3/NnTR/O8yv30NIZtPs7Q6RJz8NfOD6qkU+qUX1YzAZy7bvv9c380KGVPfLONnvbymb2y+q+6+UN1FeX2Auh5QuyaO8K8dGOSN/nPY6aSpa/CIzv5skPdlBiflfOzHZvlnu+wyeRDqYZaw7OEu89O6fdGxm+9ieECAghZTz09lZ+89KGuGPmja/h+ZV7mDG60lXKwrJhz58YvxlMKnAV+OxDolyqhEVzRzcb97WyyJOD4PTZ3Gn+3P0ExK3/XA3AlOFGGXPvU+51Dy2zNRGIaBAATW2G7+gXn5rJ/Im1ru/HKRSifBBxOrMNBpbA8vPHOH8eflNV9rjsFxEiIISUcePjH9nbl80dw5yGGr71lw+oKMpnRGUx6/a2MKqqhL9+ab5vf+fBoi/mkIEoqXDMDxYSDGsmDC2zj42tKWWrJ6MZiNty1SrR7aw+q7V2CQdwaxAhrbn4mNFccmw9XpyagVdA5KWZk7q9yzChOVvH+s2qJ2GRKGkgE3uFhLkKKcO5Nt16ydF2452xtaXc8ZljOLq+kmPHVTOnoYaKYv/qqINNsn/Xbqen++q7Xt7Af/39oz63JLUW/UNmJNjFs0dz1OghvmPjfZa18Ic13P/6Jhav2mM7pJ04TVDBkE5IAyiIE+aaDuw9bGhFvhpED9daDw0J+SAy3BAlGoQwYFhO57E1pUwZXsGTXz1xkGfkT8AVxZSsBhEby7Rz1pEjXDkfyeA0a7R2hvjY9OH88tJZ3PzEct/x8TQIi7V7DtsZ7E9ctyDu2O5Q2OVPiEU8E1M6MMR8IJk+0l+wxkPZPojMXvwTQQSEMGA4BUQsPn/ieA61RXcnG0hcJZ77pkL40uzTfS1RukORRakrFLbt/lYwwIghxex2+AwSKT3+9sYD9vb6ve6e0vkB5RIyncFwQou9V0AkIlQGkivnj2N8XRmnTnUI6gSnmMydZGoVVwsxMQkDRnVpAVfOH8fH41RkvfGC6fzi0zMHcFbROP+ok7WMxBrfFYyEjfblubMrFHbtF5kL8SizLWpTezdfOGm8fT4Ydo+//vRJPO7REpwC4N7XjCzpaaZPqLa80DW2KxhOyFzkFQjppkHk5wU4bdowXw0xUeezmJgEoR9RSvGDC48a7Gn0SCCOH6EnYo1+4I3N9nZfol+6g+4F39IgTjZNVu3dIVcZCa8CMaqqhFljquwEOmNMZJDVlOnRa+ezaNUe3tt2iD++FWkG1BUK+5ap8FLo1SDSzAfhh9937fdVJeODcL57JiIahJAS+uqIHUxceRBJ/oX4lY8G+PEzq+ztvkRHejUIS0AMrSjimhPHc+vFM+IXojNfv3XWVM6bYSQeWhrE18+YDMCEujIqSwu45Nh6vnTKRGaNqeJj04fbc0+kSF28KKZ0p8evx7yVRHqPz2uoAbB/1pmGaBBCSljnsWVnEqnQIIoLAnZ2cm/MDk1t3Vzw/16lrND9J+t8Ur/pgukA3LZwbez5mRP84ikTARj/3acJmprE5xaM52tnTKbbIYTqq0t5/LoF3P/6JhaaeReJaBDeKKZ0CG3tiVRYwSYPr0i6zHo6IQJCSAkLV+4e7Cn0mr7UYoqVKDeqsoTy4nw+3N7UKw1iZ1O7qxGPhTdjGXooZU20b8DyU6iA8aSfF8iLus5VrrsXGkQmmJh6wvrZWUI6Vf2y04nsv0Oh39l7uINnl8cWAMu2HuLRpdsYVZn60typwJ1J3T+JctrxXn7Wt7++u92OzfcjVjSSn4CI1wzHezsBFXnveI7kgEtAJKJBpHeYa1y0czP65/7tc6bytdMnccHRIwdwUoODaBBC0vx68XoefGsL9189l1OnDrOPr9tzmBsfX87bm4wmNVOHV/CzM6cwfVTyseaDiTuTOsmLY5intNb2e3md1Jv3t/Ktv3zASZPrePCa46LeckdjOwda/Rv0JK1BeBZqpZRtUop3r65+DlmQKOeHXUKjBxNgRXEB/37W1NRPKA0QASEkjdWT4bkVe1wC4pZ/rLSFA0BZUR6fnjtmwOfXnyS7rMVaBzWRhdW7/Fg1lLo9DmiAbQfbOOlnL1JbVhh1DqKjhSD+E753egEVcZrH1SAc53qVKJcBAsKPTM9j6CtiYhKSJt9cgNq6gq7jXjNIeZqWz+gJV3Ob/jIxacd7eSTEziZDQIysjO6iZ9VFiqVBFCWtQbj3Awn6W9w+iF4kymWAgKgxcz7KHBVendpEJlnJ+gvRIISEae8K8a2/fkCB+cfe2hlk+6E26quNzGhvsbiywmhnZybgWsv6yUmt0fa+NzxyV6Phexjp47M5YNZBqijO53BHMOq8rw+ilwIinuko2Z7SXhNTJoS5fvPMKdRXlXDeUdnvW0gUERBCwjy7YhdPf7jL3l+0ai+LVu3ljn89hjOOGMaOxnauXtBAZUkBv1q0LmOfuNyZ1ElqELFMTDqySFri4ZmPdlFRnG+XxigpiBaoluaQlJM6iSgm516iJqaENIj8zHNSFxfkccX8hpjnv3zqxIGbTJogAkJImJ2N/lE2jy3bbvcWmFlfZZeVCEeb1DMCdx5EcjgXYOe1ThOTpUB85U/LADhuvJFMFdawfEcTL6zey5dPnci7Ww7ZWpmffwKgMC9+SGrU/FTs/XhreH6yJqZA5pmY4nHrxTMYV1vW88AsQwSEkDC7zdaVhfkBV22h1bsP253GxtaW2uMSyTRNR1LVUS5WFVCrIU9Yay749WsAHO7o5revbrLHOIv0OelrFJPlb1EqfkhvIGkfROYlysUjAxSglCBOaiFhdpnO1GLPorSzqZ0v/GEpAONqSu0FJFOrbfSlK5xzkXXevtY65s8lZApSZ/jr855ucbFI3sTkxhKAPQlCp98hEW0gE3wOiXDB0UZhyWPH1QzyTAYH0SCEhNmwrxWItodb61pDbSk1ZYWOxSYzJUR/9YNwKlBh7VhkPZqVZYoLOY53BROzz/mFuebHC3P1SZRzvsYi0Y5wT11/Iq+s25f0zy1dOXnK0IwuldFXUq5BKKXylFLvKaWeMvfHK6XeVkqtU0o9qpQqNI8XmfvrzfMNqZ5brnCgpZPFq/a4Cui1dQWTKqi3p7mDTfsNAdEd47qbLpiOUspOjLv4mOi2lJlAf/WkdpqSjCgm/zwIS+A6f6yJCoiigj46qZVyvcbCKRTiaRBHja7kK6dOivteQuYwECamrwOrHPs/BW4JP8yMAAAgAElEQVTTWk8GDgHXmMevAQ5prScBt5njhH7gs79/h2seWMqE7z3DzsZ2OoMhpn//OVeF0Z54f1ujvR10OEydkTdW05rRVSVsvvV8zpuRmeGCfekHEUuDMJzUkW2nFmZpDk6fjVW19YZzp3HipLqYn+eXQJdcHoT7NRbJOqmF7CClAkIpVQ+cD/zO3FfA6cBfzSEPABeZ2xea+5jnz1DZoqcOMpvNJ3+ADfta6DAbtv9l6bYer/3T21vYfqiNnY2RQnHOJ9366khy19CKIrIB5ewol2w111g+CCKmq7DWLg3B0uScGl1XMMzU4RV86ZSJcU06wyqicyfi1WLyvlWiPohkw1yF7CDVPohfAd8GKsz9WqBRa21l/GwHRpvbo4FtAFrroFKqyRy/P8VzzHqcC4xCxa010x0K88LqvTz45hZeW2/86CfUlXHGEcOixpYV5vEvx4zmZ8+uAaK7j2UqiWYX++Ea7tEg7DwIDZ3BkH0u6GNi6gyG7fHxKluU+CQjxncie6KLEnVSJ5koJ2QHKdMglFIXAHu11u86D/sM1Qmcc77vtUqppUqppfv27euHmWY/3vXi9sXrY479wT9W8sUH37WFAxhRSjsa210awrCKIlb84By+cuoke/Eoys/MzGkviSaP+V4bwweBI5NaYwgAC1uD8DivbQHh+AJvumA6S288M+4cksmD6Ol45D0Te38hu0iliWkB8Aml1GbgEQzT0q+AKqWUpbnUAzvN7e3AGADzfCVwEA9a63u01nO01nOGDh3qPZ1TBENhth5oizr++Hs7aLjhadss5FzkNh9o5b7XN0VdY/Hsiugy3qGwZuvBNsY4zEnOReKlb53KI9ce36t7SEf6kvXrMjFF+SAsDULT0R3RIGwfRNhfQDjnc/aRw21fTyziRjF59q2hyZiYEinWJ2QHKRMQWuvvaq3rtdYNwGXAC1rrzwAvAp80h10FPGFuP2nuY55/QfeleW8O8Jnfvc3JP38xKsv2b8u2A7B2z2HAvWh19hAds+9wZ9Sx7pBm5c5mjh1XbR9zLhhjako5fkJt8jeQpjjXyuSL9UWI9kGY29r9PfhFMYG/BmFtzx5bxSeP9Y8Si69BxDIxxbwEcAudTCibIfQPg5EH8R3gEaXUj4D3gHvN4/cCDyql1mNoDpcNwtwyhmAobJfW7gyGXdUzoxeByHaskg09EdZw6tRhdnZvNq8RfUuUi2w7n2+01rYvSKPp7PYTEDE0CB8B8fevLIg5h0R6Ulsk7KR2PErG01CE7GJABITW+iXgJXN7IzDPZ0wH8KmBmE820Oyo7BmMsehby43zj98ZPeMVJAt7yN4dXeVvYso2+tIwyNUkyHHcGcXkdVJHwlzd75VvO6mTcxDH+268a7v1dj2VwnAnyvU4BV9qygqZ15CbGcmZijwKZCiHO7rt7Xc2H6LhhqdZtvUQEHlKPNDSxc7Gdtci1+6wfTtp6Qza5TJi4SzrkM1mhj5lUrs0iMh2OKxdDYM6HBqENS6WDyLZXgzxfAR+PamN1/jv6S610btlY9lNH+OuK47t1bXC4CACIkNx9gZYZD75v7LWHdX1rb98wAm3vuBa5No6o3sKhMOaHz+90nXsPz42JWqcW0D0bt6ZgPPW+mJicqKJPHmHtXZpEBYxTUxOjSaROkhxKwa6dyOJckmEuWbzly+4EAGRoTS3RzSI7h7qajsf+Nq6ohemtzYd4OElkaS5X3xqJtefMTlqnNPPkc0ahMsHkWyinGe87YfQXhNT9HcW5aRWlgYROdbXQnm99UGIgMhNEhYQSqkTlVJXm9tDlVLjUzctoSfcPgirGqj/2LbOiFDwExDeJ86yIv98hvyA8jV7ZBtKqYhtvo8ahEM+uBZhPw0i5BH0fj/rRARzMlFMkVpM8d/TFeaaxd+94CYhAaGUuhkj+ui75qEC4I+pmpTQM80OH4S3uqr3j93Zz9jbRxqgyNPJrLTQP3YhP0/ZC2a2V0Gx7q4v5b4h4qjWWjvCXDXr97ZEXevt+WD5EpItcxFP64nWIKzXJJzUWf7dCxES1SD+BfgE0Aqgtd5JpHyGMAg4fRDBJFq3tfpoEF7bd6lZvmH6yCGu4wWBQMJOzUwnUdOLF+9oy8SkiQiPsIZH39lOQ22pa6w3BDmgojWIvpa5iNVRLply35IolzskKiC6zKQ1DaCUyr3ee2mGM4rJ++QZ78+33SEgrMWh22MPtzSIv355Pt84M+KLCASU76KVjVj3mexdRpmYrFdPLaa2riDj69x/RkGPJpjvcVIr1ffObF6B1xsfhGgQuUOiAuLPSqm7McpkfAFYBPw2ddMSeuK1dZFaSdaTp/V3G8/84zQxWYqDV8BYPojSwnxX7gOQMyYmenmf0RqE+YrDxIR2CQyLKA3C44NIVHuIH+bq2U/QB5FsRzkhO0hIQGit/xejBPffgKnA97XWv07lxITYbDvYxtIthxhSbDzpB2P0K/bDz0ntjYJy+iAKPB3L7EUrR9aIvvsgIgEEziimsNZRT+Le79FaiG1NIsGFeVRVCT+88EhOnepTq6yXYa7J9qQWsoMeM6mVUnnAc1rrM4GFqZ+S0BNWstuFs0bz4FtbbCe19bQa78+31ZEHEcvE5CwhHSUgemmbzzSsu+u7DyLyaneU04bY8C60Xl+St9RGMv6HK+Y3sHLXYZ/59d3EJAIid+hRg9Bah4A2pVTlAMxHiMPiVXtcVVotZ3IyTmpfDcLx5PrlUydS5hAQXnNFok+c2ULyPgj/K7RZ7lsp0y+hozUCr6nPa1pK1rTjNzxWRzkxMQl+JFqLqQP4SCm1EDOSCUBr/bWUzErw5f43NgPwwbYmIPKk73VuxsNvrFPAfOecaa5zhR4NIvJUm/BHZjTJOoVj5kFoQ9goIiYmr0bg9UFYJS0CSZqYLPyEeCwfRDIaRFVpdjSGEnomUQHxtPlfSAO6QoYWYGkQ3ifPpN/PNDGdNX141DmvBpHogpKrRJmYLB8EmBqE0dHP18Tk+R7tiLEEHcle/OSJV8hYuz2ZjZznnSVXhOwmIQGhtX5AKVUIWAV61mitu+NdI6QOSyCUFFpOamOBt1vzOf7WC/MCdPVQ4tt6vx9edFTUOW9htkQXlGyhLx3lwKlBaBRGomFYW4lzHg3CYyq0hHNeL9djP3OX90iieS058nULHhISEEqpU4EHgM0Yv2NjlFJXaa1fSd3UhFhYT/wlZga0JQAi1UAjf80FeQoft4MLy8TkZ1suzHcfizzN5saKkXQUk7cWk+NVKbMnuDaEhHfhj6VB9Db3wdfEpPz3e/o+c+X7FtwkamL6BXCW1noNgFJqCvAwILV7BwHLVm0JCCv5za9oX0F+gJ4khCVwCnxMB14NQtlmjyQnnWH0Xy2mSISZAlARs1O0icnrg/CYmJKbSoy5x4piSvLNhZwgUeW1wBIOAFrrtRj1mIRBwBIQlg/CCnsNmU+gzqJv3jBVJ9aaYJmYvA5pv+v9SlBnM0lXc41Ri8k8aTupIfpn2J1AP4hk8NM8ojWI5L7PMTUlPQ8SsoZENYilSql7gQfN/c8A76ZmSoKF1pr3tjUye0yVa+GxTUymgLDaV1oRSs5IJb9F34v15OonTApihLnmismhr/0gtI5oEQpjIbbyVnrSIPoqjP0ui/ZBWK89f8ZT158YlVkvZDeJahBfBlYAXwO+DqwEvpSqSQkGT36wk4t/8wZPvL8TiCzKlsbg9UFYmoXTlu1d4J1Yo7pDYZTyf1KNlSjXW8dpptHXUhtot8agVKQ4YpQGESsPoh99ELFqMSVym0eNrqS6TEJcc4lENYh84P+01r8EO7u6KGWzEgDYtN9IOdm4z10a2kp2Ky10l+kO2iamyEKTn8BK3hXSMU1RUYlyOWdiSnK8V4NA2wLBcFJHBEZPtZi8mdTJCqtkEuVy5fsUkiPR58DFgFO3LMEo2CekkFj2b0tAlHgEhOWkDibtgwjHNEVFaxDma454NZN2Ukd1lHOEH2Ms8jFNTDGqufa2xLd/olyMvJYc0QiF5Ej016JYa20/xprbpXHGC/2INw0uokG4FUBLg3D7IHpeXIKhcMwKoLlbi6l3T+1+5b7tGlnK+B+KYWLyNn7qqznPNw9CNAghCRL91WtVSh1j7Sil5gDtqZmSsLOxnYeXbI15vt0s2V3i6QRnaQ5uH4TxFZ8xbRjDh/hbBZMyMeVYWGTyGoQbozCfZWKyopgsDSL+e/l1lEuGRDSPiA8iR75QISkS9UF8A/iLUmonxkPRKODSlM0qx/n1C+t5eMlWJg8r9z1vdYUrzA/YmbkQcXL6mZgCARWV02ARz8TkPW69RV87m2UMSWsQ0WGuzoZ9ThNTovWPeu+k9pufd0xuCXwhOeI+wyil5iqlRmit3wGmAY8CQeBZYNMAzC8nsdbkdWbfYk9HUNq7QuQFFHkBhdMqYYVJBl1O6ogd26sNWMOCoXDMaCdvdnWuPHH2X6Kc+5xSYPmie6x/pNwCwmrklCh+fqIooSQmJiEOPZmY7gasjvfzge8BdwCHgHtSOK+c5lBb/DJXrV1B3wXdzoMIOX0QxlecF1BRT/1N7d3886NddId0zGgn7yKWc2GuySbKRZXa0I4+HUbLVtvElKQGUZNkFVXfPAjRIIQk6OnPPE9rfdDcvhS4R2v9N631TcCk1E4tdznQ0unat23Y1r72j05yhrlOG1HBon8/xWViCntVEeDLf1pGVygc0wfh1RRyzanZVw0C7fj+zDBX20mdYAVV62ddk2QOQiJRTLn2fQrJ0aOAUEpZfoozgBcc5xL1XwhJcrC1y/e4c3m3NIMpwyN+CiuBLhgOM3tsFZOGldv1lQKO6BkvTe3dFCVYwtlatLLdxGSRfLE+N64oJvP9LNNeoiW2rS6AySapJeeDyI3vU0iOnlaFh4GXlVJPYEQtvQqglJoENKV4bjlLLAER9vEtPPeNk+1j729r5IXVe+gOaXtxKXD4IGI5olfubO7Rvm1FTKkcMTFZy2Wfw1wdeRDGIqzs77EnE5MVVGD9PiRrYkqsYZA5Nsu/T6F3xNUCtNY/VkotBkYCz2ttP4IGgOtTPblcJBzWUT4Iy6npzLS1TELeBezVdfs51NZFXbkR0lroMDGVF/vXV2zpDEblVDj58xfnU19t5EnmiknCzl5O8rroKCZ3JnVARfIdevoRjqs1Uo06zdpbw4cU92kufp+ZK0EHQu/o0UyktX7L59ja1ExH6AiGohKmun2ik5zawBdPmcCa3Yd5ac0+dhxqR2sYZRZVswRJnlIMKTa+7m+eOYXbFrm/wrLC2BrEvPE19nauVHPN7+V9+hfrc58P23kQ8d97khnm/LkTx9PaFeSK+eOSmou/iUl8EELiiGKZZliVWv2O+WkQAN899wjuv3oeE+rKWLa1EcCuummV4wgEoLzIEBDjh5Zx/9VzmT5yiP0eJXE0CCe5YrPOC1gaWnLX+TYMsjOpFYpIaHJPP8Ni06xXXpTPd889wt5PlERMTBLFJMRDHM1php+A6LartUYeRcuLo7+6YUOK2GgW+LM0iEi2tbIFRGtnkE/MHMWpU4cx85bnaWrvjqtBOMkVH0T/aRDaFYXm1CAsk5NHYeT2y2czoa6sV/N2kogGIT3GhXhk+Z955tHpp0HYZbwj58b7LCBOG/XISmPbqvjaFQzbQqWlI+i4xvBVlBYl9qyQlyMmiUi0VnLXRUUxaXctJmc/CKWU789x+sgKjhpdmeyUo+eShJM6y79OoZeIgEgj3tiwn398aPR+cJbytrQKpw/CT0AMqzAW+/KifNscYS38ncEQ5xw5AoBjG6rtayyh4i0dHotccWpaUWJ9jWICdzVXcGgQMcb318/Wz8chxfqEZBATU5rw21c28uNnVtn75UX5dtVWy8TkND9NHBotIGrNyKXigojcLzUFRUd3mOMm1LLxJ+e5ErSGmkIlURNTwM7uTWh4xpLX2ygmv3LfOqIxKOWt7qrw1uvtrzpXfrkt0Yly4oMQYpPlf+aZg1M4AFQ4fAx+RfiOHVeDl1ozkcpp07Y0g86gIWy82bsRDSJRJ7X1mt0rSr/5IGwPhLMWU6RYn9/C3NvifF78nNqSKCckgwiINKXCkbPQFQzT0R3isOk7uOSYevvJ34mV++AsqeHtW+1luKVBJFgILlcWlF5HMXnzILQ7kzqglJ3RbqTNRX9AfzVjcmqSkfn572e7yVDoHSkTEEqpMUqpF5VSq5RSK5RSXzeP1yilFiql1pmv1eZxpZS6XSm1Xin1obP/RC7i1CC6QmF+9uwa2rpCPPT54/jFp2f6XlNbbmoQDhXC0gwsDcKLpUEkGuaqck6DSO66KCc1xOwHYUUxeekvE1Nxvp8GISYmIXFSqUEEgf/QWh8BHA9cp5SaDtwALNZaT8ZoZXqDOf5cYLL5/1rgzhTOLe1xCoglmw5y3+ub+NSx9cyfWBvzGssHoX1NTP4axOyx1cweW8URIyqSml+2LygRM08/hLl6fA52FBP+UUz9VfaiyM/E5P0s80B/mbWE7CJlAkJrvUtrvczcPgysAkYDFwIPmMMeAC4yty8E/qAN3gKqlFIjUzW/dKeiyF0WY3RVCT+86Ki4pgDLB/Gx6cPtY5aJqaPbX4MYUVnM37+ygGFJlnHI9p7UvdcgohPlnOW+FRByCAw/+dNfGoS346D9mQ5yxWQo9I4B8UEopRqA2cDbwHCt9S4whAgwzBw2GtjmuGy7ecz7XtcqpZYqpZbu27cvldNOGV3BMHe9vIFDrV1M//6zPPPRrqgxVs5CQMGpU4dy++Wze8ykLS7I49Vvn8b/XDLDPmYtErE0iGSxSorHajCULfS2aq1/sT5HuW/ljmry1SD6y8Tk54PwSCTr/kQ+CH6kPMxVKVUO/A34hta6Oc4fnN+JqPrUWut7MJsVzZkzx79+dZrz4FtbuPWfq7n3tU20dYV8+09XlhgaxCXH1PPzT/n7HPwYU1Pq2rdCHWNpEMmyp7kDSL5wXKZh50EkeV30eO0p9+00MflrKP3npI5+oPC+da5EpQm9I6UCQilVgCEc/qS1fsw8vEcpNVJrvcs0Ie01j28Hxjgurwd2pnJ+g4VV33/fYaMx0IS6Ml5dt9815vyjRzJhaJmd3NZbKksKKC/K57/On96n97HY0WgIiPrq0h5GZja9jWLyLu7Oct9WwyC7FlPAX0NJZZirV4JZv4vigxD8SGUUkwLuBVZprX/pOPUkcJW5fRXwhOP4lWY00/FAk2WKynYa26NbjFaXFnLB0aNitgJNlPy8AMtvOZtPHlvfp/ex2G92uxtjlv/OVgrMBTNGj6WEMXwQEae0UpEoM8NJHX1Nv0UxJWBisqLXzjmqbw8iQnaSSg1iAXAF8JFS6n3z2PeAW4E/K6WuAbYCnzLPPQOcB6wH2oCrUzi3tMKvB3Vhgh3eBgsr5yJbsZ6og95Kej0Qt9y3Mkw5wbDTJ5G6KCb/MFf3/vWnT+LiY0YzZXhyUWxCbpAyAaG1fo3YJtwzfMZr4LpUzWew2XqgjfrqEl/7clNbdAe5RFuADjQPfeE4PtzelP1RTKYPwtuboyeio5i045xBWLvzIoztiCDpLw3C7zvyHikryhfhIMREajENAL9evI5fLFzLzDFVfLCtketPn+Q63+RjYspP0wX4hIl1nDCxbrCnkXIsH4SzvEkixNMgjFpMKiqzGozv2yqpkkqHsWRMC8mQno+pWcZvX90IwAfbjGY+hx3ltksK8nx9EPKHPLhYAjp5DcKNM8w1oKJbjlrPAc4GUKnUzuS3SkgGERAppKM7xN7mDpodAsE6blFTVkijjw9CGFz66oOwrjd6UkfOeYv1KYcGMRBIOKuQDGJiShFbDrRyys9fYl5DdNVVp0CoKi1gR2O7vT+2ppSTp2S/CSfd6a0GYS3AxfkBWrtC7nLf5j/tKNZnf16K6qdfNX8ca/Yc5q2NB4n6UEHoAdEgUsA9r2zglJ+/BMCSzQejzh9sjTilq0sLXecumj2aH100w3uJMMD0VoOwcOYgeMt9OzUKi1RpELdceBRXzm+w90WBEJJBBEQ/s3p3Mz95ZnXcMQcdUUtVpQVxRgqDha1BhHpXosQSEK4wV8xMakcUk/fzUoGKsS0IPSECop/53+fWUFtWyLPfOIlZY6rs484nt0NxNIhjx1UjDD6RKKbkNAgrEslKUtNmwW+IhLWGwwNnYrI+129bEHpCBEQ/s2rXYRZMqmPaiCGuJjy1ZZHEsgMOAeEs673o30/hlClDB2aiQlwsQT1txJCkrrP6blhVdL0NgwwTk7sFKUTyLlKBUyaIeBCSQZzU/ciOxnZ2NLZz+TyjpJSzjWddeaFdpsKJVW21obaUScPKB2aiQo+cf/RIZo09ndFVyZUU6TA791lZzBH9IVKLyVmszxIeBf2VPu2DUyhIFJOQDKJB9CNX3Ps2AJOGGZmpZeZTZGF+wKUpODnryBGMrirh0rljB2aSQsIkKxzAT4PQrn4QAaVcxfosUlksz21iStnHCFmIaBD9xLaDbWza3wrA6dOMFhelRcaPt7QwL2Yvh6kjKnj9htMHZpJCyrH6bhS5NIhIopzLxETExFSQwtIqIhOE3iICoh/Y3dTBST97EYB/PW6sXWjP0iDKCvPt1p9CdtNpmZgclVStah2GiSnSD8K5chekUINwaiqiQQjJICamXtLY1kVzh5Hw9sT7O+zjFUURmWuVUi4uCNhPlE7OPGJ41DEhs+kwTUyuMNeIFwIcRfmc/oCUmpgckshbTFAQ4iEaRC+Z9YOFFBcEWP3Dc115DeUOAWFpEEopO1v6rn87hqEVRZQXFTB1hFTRzDYsDSLSD9rhg7BqMflkUhekMMzV+UGiQQjJIAKiD1gRK82OYnvlDmd0qUNYfOecaby18QBnHzlCYtGzmE5bgzDzIJxJchhP8JEw18i51GoQESSKSUgGERBJEA5r/uU3r/PlU93lupvbI8X4nBqE1Vd64tAy5o2vYd746LpMQnbh1SCMjnLGOSvvwc/EVJDSPAiniUkQEkd8EEnQ0hXkg+1NfPWhZa7jli8C3IlvZx85nF9fPpv/u2z2gM1RGFy+fc5UxteVMaPeyKJ3+iCsRDkL535+CvMgAmJiEnqJCIgkaDHLdnvLLzhNTCWO5Lii/Dw+PnNUzBBXIfuY01DDi9861fY/ufIglMfE49AmUppJ7XRSi4QQkkAERBIc9vR1ACMr1tkRLqx11BghBzHXYW8mtRO3iSmVtZhS9tZCliMCwoeO7hA/fGoln7v/Hdfxls7oxj4tHUFXQ6DassKoMULuYT21R/WD8PgDIiYm6SInpB/ipPbhxseX89d3twOGY9pqAentDAew93AHze3dXHvyBE6fNoyj66uixgi5h7I1CB1ViykyxlHue4DCXAUhGUSDALqCYZejecuBVnv7kCPHocVHQHzstlcIhjXzGmo4fkJtaicqZAz2muzUIJRyOYyd26nUICS0VegtIiCA37y0nqP/+3leX78fcJfj3t8S2fbzQQD8/rNzOXO6ZEULESztwBXmCjEL56XWSS0IvUMEBLByZzMAr6zbBxgtQacMN0pvO0t0e30Q379gOs9942ROM4vzCYKFbWLS0eW+HaPsrZT6IESDEHqJ+CCI5DEcbOkiGArT1N7Ngkl1rN3Twqpdzaza1cxJk4e6WokW5gX43InjB2vKQppjLckad7lvV5SrS4OQKCYh/RABARwwzUgHW7tobO9Ga5g6vIKn2cWPnl5ljlplj7/3qjmMqy0bhJkKmYJLg3CU1nA+zbvCXCWKSUhDREAQMSMdaO1iy4E2ABrqypjbUM27Ww7xo4tmsL+lkzkN1ZQW5rt6TQuCPw4fhOOI8oyoKCoA2gesJ7UgJEPOC4hgKMyhNsPEtGb3Ya79w1IAxlSX8KfPH8+e5g7G1JQO5hSFDCSiQURMTCiiTEwjK4tZuas5xR3lUvbWQpaTkwKiqb2bTftbmTq8gmVbDwFGNc32bqMS54PXzGP2WKNpvQgHoTdEfBDOWkzKVfYioBQjKosBaOn0j5Drz7kIQrLkpID43asb+fUL6+39wrwAp0wdysKVe/j+x6dz0uShgzg7IRtQkUw528aklLu7GxgaBMC+w52kCjExCb0lJwXEF0+ZSFlRPu1dIapLC5g2cggTh5bz8Zmj+PjRIwd7ekIW4IpiMrcDyq1BKAUThxrh1Ckt952ydxaynZwUEOVF+XzplIlRxz8xc9QgzEbIRpxRTK4GQa5MasU5R43gp5fM4OQpQ3l4ybaUzEUyqYXeIolygpAC3MX6rGPeWkyG+efSuWMpK0rds5rIB6G3iIAQhBTg44KI6gehBiiTWhB6iwgIQUghRphrJM5VuUxMzm0JcxXSDxEQgpBCvBqE18RkkdI8CHFTC71EBIQgpACnk5oY1Vyd4iIvhY/5KWx3LWQ5afWro5Q6Rym1Rim1Xil1w2DPRxB6i/IJdFUqjokphRpEWWFOBisK/UDaCAilVB5wB3AuMB24XCk1fXBnJQi9w12szzwGnjyIgTH91JZLG1yhd6SNgADmAeu11hu11l3AI8CFgzwnQegVrigmRya1qxbTAM2lVDQIoZek02/OaMCZKbQdOG6Q5iIIfcLSFH776ka2mhWCjUxqxxjxHQtpTjoJCL8/Fx01SKlrgWsBxo4dm+o5CUKvGFNTwvwJtby58YDruLO1qDe09dNz6jnjCGldK6QP6WRi2g6McezXAzu9g7TW92it52it5wwdKkX1hPSktDCfP1wzz3VMKciPE1L0s0/O5OwjR6R6aoKQMOkkIN4BJiulxiulCoHLgCcHeU6C0Gu82dEKRYGjMdBAmpgunDWK6tKCgftAIStIGxOT1jqolPoq8ByQB9yntV4xyNMShF7jjVJSyl21dSCL6P3fZbMH7LOE7CFtBASA1voZ4JnBnocgpMXQesEAAAhoSURBVAJDQAyOBiEIvSGdTEyCkNUolMtJLSUwhHRHBIQgDBBeDUIKuArpjggIQRggFJ7OcSIghDRHBIQgDBBRPgiREEKaIwJCEAYIpRT5YmISMggREIIwQCigMG/gi/UJQm8RASEIA4RSypVJLeJBSHdEQAjCAKGAgnyniUlEhJDeiIAQhAFCKSgISBSTkDmIgBCEAWIwazEJQm8QASEIA0RYa08mtSCkNyIgBGGA6A6FKcwTH4SQOYiAEIQBoisUduVBiHwQ0h0REIIwQHSHtKvUhmRSC+mOCAhBGAAunzeGGaMrxUktZBQiIAQhhcxtqAbgxxfNIC8gUUxCZpFWDYMEIdu497Nz2bK/jYCZ/yD9IIRMQjQIQUghQ4oLmFFfae8XSrE+IYMQASEIA0i+FOsTMggREIIwgLj7QQhCeiMCQhAGkMJ8cVILmYM4qQVhABlSXMB/nj2VUFiLiUlIe0RACMIAc91pkwZ7CoKQEGJiEgRBEHwRASEIgiD4IgJCEARB8EUEhCAIguCLCAhBEATBFxEQgiAIgi8iIARBEARfREAIgiAIviit9WDPodcopfYBW3p5eR2wvx+nk05k871Bdt9fNt8bZPf9ZdK9jdNaD+1pUEYLiL6glFqqtZ4z2PNIBdl8b5Dd95fN9wbZfX/ZeG9iYhIEQRB8EQEhCIIg+JLLAuKewZ5ACsnme4Psvr9svjfI7vvLunvLWR+EIAiCEJ9c1iAEQRCEOOScgFBKnaOUWqOUWq+UumGw59MblFL3KaX2KqWWO47VKKUWKqXWma/V5nGllLrdvN8PlVLHDN7Me0YpNUYp9aJSapVSaoVS6uvm8Wy5v2Kl1BKl1Afm/d1iHh+vlHrbvL9HlVKF5vEic3+9eb5hMOefCEqpPKXUe0qpp8z9bLq3zUqpj5RS7yullprHsuJ304+cEhBKqTzgDuBcYDpwuVJq+uDOqlfcD5zjOXYDsFhrPRlYbO6Dca+Tzf/XAncO0Bx7SxD4D631EcDxwHXmd5Qt99cJnK61ngnMAs5RSh0P/BS4zby/Q8A15vhrgENa60nAbea4dOfrwCrHfjbdG8BpWutZjpDWbPndjEZrnTP/gfnAc4797wLfHex59fJeGoDljv01wEhzeySwxty+G7jcb1wm/AeeAD6WjfcHlALLgOMwEqzyzeP27ynwHDDf3M43x6nBnnuce6rHWCRPB54CVLbcmznPzUCd51jW/W5a/3NKgwBGA9sc+9vNY9nAcK31LgDzdZh5PGPv2TQ5zAbeJovuzzTBvA/sBRYCG4BGrXXQHOK8B/v+zPNNQO3AzjgpfgV8Gwib+7Vkz70BaOB5pdS7SqlrzWNZ87vpJdd6Uvt1ic/2MK6MvGelVDnwN+AbWutmpfxuwxjqcyyt709rHQJmKaWqgL8DR/gNM18z5v6UUhcAe7XW7yqlTrUO+wzNuHtzsEBrvVMpNQxYqJRaHWdsJt6fi1zTILYDYxz79cDOQZpLf7NHKTUSwHzdax7PuHtWShVgCIc/aa0fMw9nzf1ZaK0bgZcwfC1VSinrgc15D/b9mecrgYMDO9OEWQB8Qim1GXgEw8z0K7Lj3gDQWu80X/diCPd5ZOHvpkWuCYh3gMlmVEUhcBnw5CDPqb94ErjK3L4Kw3ZvHb/SjKg4Hmiy1OF0RBmqwr3AKq31Lx2nsuX+hpqaA0qpEuBMDIfui8AnzWHe+7Pu+5PAC9o0aKcbWuvvaq3rtdYNGH9bL2itP0MW3BuAUqpMKVVhbQNnAcvJkt9NXwbbCTLQ/4HzgLUYdt//Guz59PIeHgZ2Ad0YTynXYNhuFwPrzNcac6zCiNzaAHwEzBns+fdwbydiqOEfAu+b/8/Lovs7GnjPvL/lwPfN4xOAJcB64C9AkXm82Nxfb56fMNj3kOB9ngo8lU33Zt7HB+b/Fdb6kS2/m37/JZNaEARB8CXXTEyCIAhCgoiAEARBEHwRASEIgiD4IgJCEARB8EUEhCAIguCLCAghJ1FKhcyKnNb/uJV9lVJfUkpd2Q+fu1kpVdeL685WSv23UqpaKfVMX+chCImQa6U2BMGiXWs9K9HBWuu7UjmZBDgJI+HsZOD1QZ6LkCOIgBAEB2aZiEeB08xD/6q1Xq+U+m+gRWv9v0qprwFfwihNvlJrfZlSqga4DyOZqg24Vmv9oVKqFiOxcShGMphyfNa/AV8DCjEKEn5FG3WanPO5FKPq8ATgQmA40KyUOk5r/YlU/AwEwUJMTEKuUuIxMV3qONestZ4H/D+MWkJebgBma62PxhAUALcA75nHvgf8wTx+M/Ca1no2RumFsQBKqSOASzGKv80CQsBnvB+ktX4UOAajtPsMjOzr2SIchIFANAghV4lnYnrY8Xqbz/kPgT8ppR4HHjePnQhcAqC1fkEpVauUqsQwCV1sHn9aKXXIHH8GcCzwjlmptoRIkTcvkzHKNQCUaq0PJ3B/gtBnREAIQjQ6xrbF+RgL/yeAm5RSRxK/tLPfeyjgAa31d+NNxGxrWQfkK6VWAiPNXhLXa61fjX8bgtA3xMQkCNFc6nh903lCKRUAxmitX8RojFMFlAOvYJqIzF4I+7XWzZ7j5wLV5lstBj5p9hWw+hqP805EG20tn8bwP/wMo0DcLBEOwkAgGoSQq5SYT+IWz2qtrVDXIqXU2xgPUJd7rssD/miajxRGr+VG04n9e6XUhxhOaqv88y3Aw0qpZcDLwFYArfVKpdSNGN3JAhiVea8DtvjM9RgMZ/ZXgF/6nBeElCDVXAXBgRnFNEdrvX+w5yIIg42YmARBEARfRIMQBEEQfBENQhAEQfBFBIQgCILgiwgIQRAEwRcREIIgCIIvIiAEQRAEX0RACIIgCL78f/elHdT2taUEAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x143da6d8>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(1, len(scores)+1), scores)\n",
    "plt.ylabel('Score')\n",
    "plt.xlabel('Episode #')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
